> Democratizing how we access networks through a universal Router and Open source software.
> Our vision at VyOS is to dramatically change how we access networks so that we can all build the solutions we always dreamed of, without restrictions, limitations, or prohibitive costs.
But I'm still kind of none the wiser. Does this thing use Linux or something else?
It's a fork vyatta[1], the same system used by Ubiquiti for EdgeOS.
Yes, it's Debian with a declarative configuration system. It works more or less similarly to NixOS, if you know about that: basically it uses a bunch of perl scripts to install and set up software from a single unified configuration file, hiding all the implementation details.
And it works great because your router starts from a read-only image, then executes the scripts and applies the configuration. You can easily roll back and forward configuration and firmware versions.
It's a Network Operating System for configuring routers & switches (implemented on top of Debian). This means:
* You can SSH in and configure it like you would a managed switch or router. There's a single object that models all of the device's configuration, when you commit it a bunch of scripts activate and actaully apply the configuration to the running system.
* Deployments are image based, you can roll back to older images etc.
* You don't need to look at any of the system's underlying configuration files or use any of the normal Linux commands to examine and manipulate the state of the system (the commands are still there for convenience of course). You don't even need to be aware that you're really using a bunch of custom bash functions to examine and manipulate the state of the system.
It's a Debian 8 base with a lot of its own custom packages. The homepage has an FAQ that tries to reassure you that being based on Debian 8 is not a problem. Whether that convinces you or not is up to you, of course.
I personally use regular Debian 12 on my router without problems. It also has "declarative config" since all the configuration, firewall rules, etc are a bunch of config files that I can scp / ansible over any time.
In 2011, Ubiquiti launched their EdgeMax products with EdgeOS which was a fork of Vyatta Core 6.3 ported from x86 to Cavium.
In 2012, Vyatta was acquired by Brocade.
IN 2013, Vyatta Core 6.6 was forked as VyOS.
That's the rough origin of these three OSes.
I used Vyatta Core on a PC at a startup from 2009-2013 as our office router. I haven't paid attention to it or VyOS since then.
I've been running various EdgeOS routers at my home since 2014 or so, first an EdgeRouter Lite and today an EdgeRouter 4.
EdgeOS has been updated quite a bit over the years from its Vyatta Core origins, but the original developers are no longer with Ubiquiti. EdgeOS hasn't seen updates in quite some time now.
Also, not all Ubiquiti run EdgeOS. Only the EdgeRouters do. The rest of their products run a completely different OS, generally either UbiquitiOS or UnifiOS.
I was pretty confused what it is too and then I loved it.
It's debian plus some shell trickery and CLI tools that let you configure debian and debian packages as a router from one large config tree using neat CLI tools (that support commit/rollback).
Normally you'd need iptables, a separate DNS package, DHCP server, etc etc to set up a router, with VyOS you just change VyOS config and it configures normal debian packages for you.
Plus everything is exhaustively tested and configs are reverse compatible, hiding all breaking changes underneath.
It's super neat and it works perfectly on a £100 fanless Celeron J4125 box from Aliexpress as a home router, routing and shaping 1gbit without breaking a sweat and with deeply sub-ms delay.
Do you have an idea why the CLI tools aren't distributed independently? Why shouldn't I be able to run it on a Debian system I already have (and understand)?
Running an entire new distro just seems like overkill for what it actually does over a normal Linux system. It's just a configuration manager!
I don't know, but I imagine it would make testing substantially more difficult (right now they have fully tested images available), and it would be impossible to implement their distro update thing (you can update the entire .iso backwards and forwards, the config will be reapplied).
VyOS is unfortunately completely useless for larger applications, since it's difficult to impossible to automate due to it's unique way of applying configurations. Don't get me wrong, for manual administration it's great, there's a lot of missed automation potential given that it's just Linux underneath.
As an example, the Ansible modules for VyOS are basicially just variations of an adapted ansible.builtin.shell, instead of offering to manage state in a more first class manner (via attributes and values):
From what I've seen of VyOS using a configuration file that is then used to generate the actual system configuration I'm not really sure its so hard to automate. Take your target state, generate a configuration file in the right format, then send it over and apply.
To be honest this feels more like a limitation in Ansible, which has always felt like a bit of a hacky config management system to me in that the way it functions is generally to run a bunch of commands that gradually mutate the system's state, rather than atomically applying the target state, but then I've been spoiled by NixOS on my personal infrastructure recently.
UniFi doesn't use VyOS anywhere afaik. Some Ubiquiti products run EdgeOS which is a Vyatta fork (EdgeRouter, the original security gateways). More recent UniFi products (UDM, UDM-SE, UNVR, etc) run UniFi OS which is their own Debian-based OS that runs applications inside Podman containers.
AT&T had bought Vyatta before selling it to it's current owner, but I know they used a REST API internally when deploying it for 5G Edge use cases. It looks like VyOS gained an API in 2019.
The theory of what VyOS does is (per my understanding) really simple: Configure all the networking components of a Linux system from a single place.
Why isn't doing this much more popular? All the systems are already there, after all! Why aren't there (that I know of) dozens of projects to accomplish this relatively easy, but relatively useful task?
I think it's a pretty big deal to be able to configure that stuff from a single place. Commercial router manufacturers all do it. Why does (as far as I know) only VyOS do it on the open source side of things?
IMO, Open Source routers are a niche thing, and open source declarative CLI routers even more of a niche thing.
Most enterprises prefer buying something with a support contract from a known name vendor (Cisco, Juniper, etc.). Most home users just use what their ISP provides them with, and of those that want something more, they either pick a SOHO vendor like Ubiquiti/Mikrotik, or if DIYing the hardware, choose pfSense / OPNSense / DD-WRT for the clickOps options, because networking really isn't trivial. For those for whom networking is trivial, Debian is fine router OS if you know your way around iptables and friends.
That leaves all those who want to use DIY hardware, and a enterprise-like declarative CLI. That's really not a whole lot of people in the end.
The question 'what is this thing' is probably best answered by the Github project page: https://github.com/vyos
It's a decent-ish option if you need advanced routing functionality; one thing to keep in mind, though, is that unless you're OK with running unstable 'nightly' code, you'll be spending USD 8K+ on an annual basis.
> If you are an individual, you can get the generic ISO by donating on Open Collective. And if you are contributing to VyOS, whether you are writing code, improving the docs, or promoting VyOS publicly, we are happy to share pre-built images with you through contributor subscriptions. Finally, you can always build your own images — just follow these instructions.
Sounds fair to me. Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.
> Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.
OpenWRT comes to mind, I've been using it for decades on first dedicated hardware, the last 6 years running in a container on a ProxMox box (DL380 G7). It has no problems whatsoever routing at (gigabit) line speed using a few megabytes of RAM and a few cores. Configuration is mostly declarative using UCI although it also offers the freedom (which comes with responsibility) to use scripts. I use the latter to deal with edge cases which lie outside of the purview of normal routing operations, e.g. triggered actions related to the use of Timelimit [1] on my daughter's phone, IoShit things with special needs, etc.
If you want a web GUI, then pfSense or OPNSense are the general go tos.
However, if you're comfortable with CLI and modifying configs in /etc/ then just running a bare metal Alpine Linux box is perfectly doable on a tiny box. iptables/nftables for firewall/NAT, dnsmasq/bind9 for dns, dnsmasq/isc-dhcp for DHCP. I've got a handful of these boxes all interlinked via wireguard, sharing routes via BGP using bird.
Sure, you miss the config verification that VyOS provides, but does mean you learn the underlying tools themselves and that knowledge is portable to any other box running those systems.
Personally, I don't quite understand why VyOS is a standalone distro when it could just be a config generator/checker package. Could even support multiple different underlying tools so if you want to use dnsmasq over bind9, or vice versa, it can provide a unified config interface for them.
isc-dhcp is EOL. I'd suggest using kea-dhcp from the same ISC. I believe there could be a script or some kind of migration path from isc-dhcp to Kea. I've been using Kea in production with no problems.
I really wish there was a better option for a Linux DHCP server. At my past job we were using isc-dhcp and it was absolutely horribly showing it's age (md5 api "keys", bespoke socket-based API "protocol", most things impossible to do via the "API", clustering that didn't really work, etc.). Kea is barely an improvement, with oddities such as being written in C++ and requiring recompiling for plugins, or an extremely weird API. It's obvious is still written by the same old folks who have no idea how software is supposed to work in the 2020s.
We need something modern - easy clustering, modern API, event stream, gRPC-based plugins, etc. (And yes, I have thought about developing it myself, it's on my pile of TODO)
One remark.
Kea doesn't have some of functionalities provided by ISC-DHCP.
If you use a lot of dhcp-eval and make decisions based on different dhcp options content kea is still a no-go since there is no workarround. Usually at ISP level dhcp is one puzzle of much more complicated system.
I tried if few times and every time I stuck on something and messagefrom developers was: this isc-dhcp feature is not supported.
This was huge national scale ISP and bypassing those limitations means a lot of $ to adapt surrounding systems providing input to isc-dhcp LDAP DB in its own config style.
If wanting internal and external subnets as "zones", iptables/nftables lets you match against incoming and outgoing interfaces. It would be trivial to make match against an incoming interface and jump to a zone specific chain. This is how I manage private subnets. fw-mark is also useful for setting routing rules. Can change which routing table is used by matching rules in iptables.
If wanting to do more stateful things, I'm not aware of any default package, but setting a rule to send packets to an NFQUEUE and implementing some custom logic on that nfqueue would be rather trivial too. I'm sure eBPFs are useable in there somewhere too, but I've very little experience with them.
Obviously iptables/nftables has its own issues, as seen in recent (and not so recent) posts about it being bypassable with raw sockets, but that tends to be host only and not when used as a gateway.
You create a _zone_. You name it and assign some interfaces to it. For my needs, I only assign 1 interface per zone. Then, you specify with which other zone that zone can receive traffic from. That also comes with the identification of a firewall rulesets to apply to that pair.
So, `'Zone WAN (iface eth0) <- Zone LAN (iface eth1)' => apply fw LAN-TO-WAN`
When you do that, the firewall rules become much simpler to write and maintain.
But, a best practice is to assign every zone to every other zone. This soon becomes a combinatorial nightmare. When you want to add a zone, you have to create 2xN new zone configurations and 2xN new firewall rulesets.
iptables -N eth0toeth1;
iptables -P eth0toeth1 DROP;
iptables -A FORWARD -i eth0 -o eth1 -j eth0toeth1;
iptables -A eth0toeth1 -m tcp -p 80 -j ACCEPT;
# add any more rules
Or, as you say to avoid exponential combinations, just make a chain for each zone (interface) and explicitly allow specific protocols/ports to target interfaces. Zones with multiple interfaces are just multiple rules to jump to the same zone chain.
You have very narrow definition of what software router/service router is. Firewall and two interfaces and VPN server is not the best scenario :)
All those systems (pfSense etc) are for private/soho use.
Big networks need stuff that is not avaiable on mentioned platforms like i.e BFD (Bidirectional Forwarding Detecion), MPLS (MultiProtocol Label Switching), VXLAN (Virtual Extensible LAN), IS-IS routing protocol or Segment Routing.
If anyone knows other Opensource routing software that support all of this - let me know. To my knowledge vOS is the only one.
DANOS [0] supports all of those I believe. It's essentially AT&Ts continued development of Vyatta (a subset of it at least) with a DPDK dataplane + multiple other enhancements.
Definitely they are not. I couldn't find any other open source routing software that has support for following very popular features on enterprise level:
- MPLS
- VXLAN
- IS-IS
This is must have to be considered by any ISP/enterprise where networking is their core business.
vOS has all three. More or less buggy but they are here.
A definite tangent: About three house moves ago I had OpenWRT on a cable router and knew what was going on. But with family and work and house moves I am now just staring at a flashing BT Home Hub and wondering how to tackle the inevitable "of course that router firmware wont allow that and your ISP wont give out its password and ..."
Is there a uptodate reliable guide (possibly including how to persuade your wife it's a good idea to drill holes in the living room ceiling to run cat6)
Same here. And AFAICT those "Hubs" have no bridge mode, so the best you can do is double-NAT yourself. Even if you can replace it with your own device, I've just received a letter informing me that they're migrating our landlines to be VOIP, delivered through the phone socket on the back of the router, so if you want to keep landline service then you may need to keep their hardware too
What I find easy(ier?) is to run (x)sense on a dedicated firewall and either a mesh with cheap openwrt routers or get something like Deco mesh and run it in AP mode if you don’t have cat6 at home. I think this combination can be under 300$ for a 3-pack of mesh Deco x20 + an intel card on a refurb dell optiplex.
IME (x)sense is quite problematic when you start doing anything other than the bare minimum. I spent multiple days trying to figure out why it silently stopped accepting IPv6 delegation, or why does it spike latency for no visible reason under load. The underlying reality is that FreeBSD's network stack is much more conservative and has less resources than Linux's, which shows up in articles like this one [1].
On the same Celeron J4125/i226 box VyOS was absolutely perfect, not a single issue, significantly low (and always low) latency with higher throughput.
On the hardware side, I think the /r/homelab hivemind doesn't get challenged enough. Dell optiplexes cost very similarly to Aliexpress Protectli alternatives (such as [2]), while being larger, having a fan, and being overall more hassle. TP Link/Ubiquiti WiFi APs seem to be overall inferior to Aruba Instant On, which is exactly the same hardware HP sells in their Aruba line, but for the same SOHO price.
The biggest reason I like xSense is because of unbound and using root dns. OpenWRT supports that, but otherwise I have to setup a local linux instance to serve DNS via unbound. With pfsense and opnsense I find that the dns is a lot more stable than unbound as using a resolver seems to work more reliably than a forwarder like dnsmasq.
I like the idea of VyOS. Networking software tends to have so much hidden, hard to control state for reasons I don't understand. As a programmer this melts my brain.
VyOS finally lets you have all your configuration in one, easily controlled place. Nice!
For some reason I ended not actually trying it out too actively. I think I was weirded out by the distribution model and concerned by the small community.
The idea of VyOS is awesome: have your configuration all in one place, apply it atomically.
After using it for several years, the implementation is clearly lacking. It seems that the maintainers are overloaded, because contributions to fix minor issues, or add config options get ignored in my experience. As a result, the configuration is missing some nice options in the IPv6 space (so-called tethering), and it's still using iptables for packet filtering. It's also rather hard to roll your own, with your own modules: the module system is rather hard to use.
Not to mention the lack of interest to roll out a version for ARM.
My next router will be based on NixOS, and will attempt to recreate the awesome UI of VyOS.
Unfortunately they went full "Vyatta way" or "RedHat Way" by basically giving rolling release for free only. I remember times when Vyatta went behind paywalls Vyos was completely free.
Not sure who would want something rolling on device like router.
Nowdays everyone wanting something good and free go OpnSense way.
The biggest problem I have is that they only support (or at least that's what they release publicly) x86_64. I am forced to use openwrt because the vast majority of consumer low-power hardware is using ARM or exotic architectures. VyOS interface is vastly superior, though.
VyOS is not targeted for such usage scenarios. It is ISP who are paying for support. I have VyOS systems with BGP, MPLS, PPPoE termination, etc that are pushing gigabits of traffic. This is VyOS usecase. Not 5W router handling home or small office.
I've used EdgeOS in the past and it's perfect for SOHO routers too, but unfortunately is half full of proprietary extensions and semi-abandoned. I think VyOS could take its place nicely, openwrt feels like a toy compared.
It's been a while since I tried it, but I remember this roughly being my experience as well.
I'm usually all for up to date software, but on my networking equipment??? I don't really want to beta test that stuff, but that's what they seem to want to make me do.
It is not really beta software though. If you don't want to go into the trouble of building your own ISO, then, yes, you are a beta tester because the only thing they make available pre-built is the ISO from the 1.4 branch which is in flux.
You can build the ISO from the LTS branch though and that branch doesn't move much. Though, I don't know how you can tell which commit was used to release, say v1.3.2. For the moment, I simply build an LTS ISO using the latest commit of the LTS branch. That strategy has been rock solid for years now.
I thought this would be about building VyOS from source, but fun article nonetheless.
I use VyOS whenever I need layer 3 routing in vSphere for test environments. NSX-T is (way) faster by dint of being deeply integrated into VMkernel, but VyOS is pretty performant for what it is and is easier to install to boot.
I was wondering if this was Vyatta... it seems it is the community continuation after Broccade ceased development. Used to run this over a decade ago on a 'router' (ThinkCentre Tiny), but eventually went to a Fedora installation a few years back. Might have a look again.
... have to say that the offering is confusing; it is a subscription or a rolling release.
Wondering how much can be automated of the installation/state; would it be possible to use version control? If not, I can see the appeal to suggest Nix over this.
I just finished building a router based on NixOS and I must disagree.
NixOS modules mostly targets desktop and servers, but the network specific configuration is still very lacking. A few examples:
- up until a couple of weeks ago the hostapd module was basically a toy: could only manage a single SSID, no way to configure the radios, hardcoded to WPA2-PSK;
- the NixOS firewall is still based on iptables and conflicts with nftables, so you must disable and manually write rules;
- the `networking.nat` module (NAT44) doesn't do NAT reflection;
- I had to write a module for Jool (NAT64, SIIT);
- I had to write a module for libreswan (IPsec);
- I had to write a module for automatic rollbacks, otherwise you can loose access if you make a mistake.
Vyatta and VyOS also provide a much higher level abstraction over the software that is being configured (e.g. you don't have to deal with a specific IPsec implementation). Finally, once you do `nixos-rebuild switch` you're on your own, while with vyatta you have a clean command line interface to inspect the state of the router and manage it.
And this is just basic routing and firewall with NAT.
Now try to setup 20 MPLS L3-VPN's + dozen VPLS services on top of IS-IS routed network with that ;-)
Well, I actually enjoyed contributing to NixOS, but running it on a router is certainly not a "it works out-of-the-box" experience.
Even if you are building a simple SOHO router you need to know a lot more about the implementation details (which daemon does what, how to write firewall rules, how to setup the network interfaces, etc.). This is because NixOS is general-purpose OS, unlike VyOS or openwrt.
does vyatta/vyos store the configuration in a single file or easy to back up config? I do not remember anymore how this was done, but I believe that is the appeal what 'nix' was mentioned.
https://vyos.io/
The VyOS website says:
> Democratizing how we access networks through a universal Router and Open source software.
> Our vision at VyOS is to dramatically change how we access networks so that we can all build the solutions we always dreamed of, without restrictions, limitations, or prohibitive costs.
But I'm still kind of none the wiser. Does this thing use Linux or something else?