We'll be migrating the CI server (https://ci.vyos.net) to a different host this weekend. It means over this weekend there will be no automatic package or image builds for a time.
We hope this will not take too long, but in the worst case we will have it working by Monday.
Nothing else will be affected by this maintenance so the overall impact on the end users should be very low.
Recently we have merged the old VyOS product on Amazon (which was free as in price) with the new one listed by Sentrium (reminder: the company setup by VyOS maintainers to provide commercial services for VyOS) that is available at ~$60/year as a means to support the project. It’s probably a good time to talk about our cloud platform support strategy.
It’s quite obvious that cloud platforms are popular, and that VyOS is often used as an alternative to their own VPN and routing facilities or proprietary products, because of both functionality and price. So far we’ve been providing our official EC2 AMI for free. This approach is not exactly sustainable since we ourselves do not benefit from it in any way, but the support level required of an AMI listed on the marketplace creates work that has to be done, and so far (past the initial contribution of the person who wrote the first AMI build scripts) it has been done solely by the maintainers.And, let’s face it, VyOS needs funding sources. There are many activities that get virtually no community participation, including documentation writing, code refactoring, developing automated tests and so on. We are grateful to everyone who contributes, but that’s not enough to keep VyOS as a service provider and enterprise quality router. Migration to Jessie was the point when making an enterprise quality router OS by few people in spare time stopped scaling, and now, at least until we refactor the whole system to be easy to maintain and extend, either it's going to take a very long time to get done or we need a way to free the time of the maintainers and hire some community members (or someone else) to get it done at reasonable pace.
Some companies mentioned a possibility of corporate sponsorships but those offers failed to materialize, donations likewise have been enough only to pay for domains and some hosting. Switching to a RedHat-like model with paid access to LTS releases is something that we are considering, but not until a stable 1.2.0 release is made. As for now, an alternative revenue stream is needed to break the cycle, and making the official cloud images paid looks like the most viable option. This wasn't an easy decision, but it had to be made.
We've been keeping both the old (free as in price) 1.1.0 listing and new paid listing updated to 1.1.8 active for a while and so far the new one has been positively received and is already bringing us a small profit. We hope most of the old AMI users will choose to support the project as well, but if not, you have time until February 2018 to migrate your setup to self-built or community AMIs.
VyOS will remain free as in freedom forever — the goal of the project was and is to make a free and open source router that people can study and modify. It will always be possible to build a cloud image yourself and use it for free. While we hope most cloud users who run unmodified images will choose to support the project, trying to prevent people from building their own image would be against the free software ideals. We are also not going to hide the build scripts we use for building the official images, so self-built images will be identical to the official ones — without the convenience of deploying them in one click from the marketplace.
The AMI build scripts for 1.1.x can be found at https://github.com/vyos/build-ami
The build process is fairly straightforward and boils down to "./build-ami $isoURL". Right now it only takes signed releases and ourselves we test new images by upgrading already deployed instances to them, but we may add an option to allow building AMIs from unsigned images in the future.
James Brown reported on phabricator that LLDP is not working in 1.1.8. Quite a mess up on our side: the reason it's not working is that it's built against the old OpenSSL 0.9.8 which is no longer in VyOS, but since the debian/control was missing dependency on libssl, it wasn't detected as dependent on OpenSSL and thus wasn't rebuilt.
If you want LLDP back right now, you can install the helium3 package from our repository (http://dev.packages.vyos.net/legacy/repos/vyos/pool/main/l/lldpd/).
One feature is missing from the changelog because it was committed "stealthily", without a task number, and thus I missed it. It's the command for setting up route-map for installing routes imported from OSPF.
set protocols ospf route-map MyMap
This route-map can prevent routes from getting installed into the kernel routing table, but make no mistake, it is not incoming route filtering (which would be very bad for OSPF). The routes will stay in OSPFd and in the RIB (they will be displayed as inactive).
So far it looks like we are going to make an 1.1.9 release to fix the LLDP bug. Perhaps we should also cherry-pick something safe from 1.2.0 too, if you have any specific bugfix or a tiny feature is mind, let us know.
The official 1.1.8 AMI passed the automated tests and it's on its way to the marketplace, the manual review by the marketplace team will take perhaps a week or so.
If you want to make you own, you can already use the AMI build scripts and point it to the (signed) release image URL.
I've re-enabled the old patch for IPv6 VRRP in the current branch and it will be in today's nightly build. In 1.1.x, we had to disable it because in the older keepalived version IPv4 and IPv6 VRRP were mutually exclusive, however, in the current branch, it seems to work. If you feel adventurous, please test the nightly build (on lab VMs!) and tell us it it works for you.
Also, on the forum, it was reported that the current branch image doesn't build. I've resolved that problem today so if you want to build an image, it should work.
1.1.8, the major minor release, is available for download from https://downloads.vyos.io/?dir=release/1.1.8 (mirrors are syncing up).
It breaks the semantic versioning convention, while the version number implies a bugfix-only release, it actually includes a number of new features. This is because 1.2.0 number is already assigned to the Jessie-based release that is still in beta, but not including those features that have been in the codebase for a while and a few of them have already been in production for some users would feel quite wrong, especially considering the long delay between the releases. Overall it's pretty close in scope to the original 1.2.0 release plan before Debian Squeeze was EOLd and we had to switch the effort to getting rid of the legacy that was keeping us from moving to a newer base distro.
You can find the full changelog here.
The release is available for both 64-bit and 32-bit machines. The i586-virt flavour, however, was discontinued since a) according to web server logs and user comments, there is no demand for it, unlike a release for 32-bit physical machines b) hypervisors capable of running on 32-bit hardware went extinct years ago. The current 32-bit image is built with paravirtual drivers for KVM/Xen, VMware, and Hyper-V, but without PAE, so you shouldn't have any problem running it on small x86 boards and testing it on virtual machines.
We've also made a 64-bit OVA that works with VMware and VirtualBox.
Multiple vulnerabilities in OpenSSL, dnsmasq, and hostapd were patched, including the recently found remote code execution in dnsmasq.
Some notable bugs that were fixed include:
A number of VPN providers (and some corporate VPNs) require that you use user/password authentication and do not support x.509-only authentication. Now this is supported by VyOS:
set interfaces openvpn vtun0 authentication username jrandomhacker
set interfaces openvpn vtun0 authentication password qwerty
set interfaces openvpn vtun0 tls ca-cert-file /config/auth/ca.crt
set interfaces openvpn vtun0 mode client
set interfaces openvpn vtun0 remote-host 192.0.2.1
A few OpenVPN options that formerly would have to be configured through openvpn-option are now available in the CLI:
set interfaces openvpn vtun0 use-lzo-compression
set interfaces openvpn vtun0 keepalive interval 10
set interfaces openvpn vtun0 keepalive failure-count 5
set interfaces vxlan vxlan0 address 10.1.1.1/24
set interfaces vxlan vxlan0 remote 203.0.113.50
set interfaces vxlan vxlan0 vni 10
The as-override option that is often used as an alternative to allow-as-in is now available in the CLI:
set protocols bgp 64512 neighbor 192.0.2.10 as-override
set policy route-map Foo rule 10 action permit
set policy route-map Foo rule 10 set as-path-exclude 64600
set system flow-accounting buffer-size 512 # megabytesThere are a few more options for NetFlow: source address (can be either IPv4 or IPv6) and maximum number of concurrenct flows (on high traffic machines setting it too low can cause netflow data loss):
set system flow-accounting netflow source-ip 192.0.2.55
set system flow-accounting netflow max-flows 2097152
set interfaces ethernet eth0 vif 42 egress-qos 1:6
set interfaces ethernet eth0 vif 42 ingress-qos 1:6
set system sysctl custom $key value $value
set interfaces ethernet eth0 dhcpv6-options duid foobar
Under "set interfaces ethernet ethX offload-options" you can find a number of options that control NIC offload.
Now you can specify options for the *.* syslog pattern, for example:
set system syslog global facility all level notice
I would like to say thanks to everyone who contributed and made this release possible, namely: Kim Hagen, Alex Harpin, Yuya Kusakabe, Yuriy Andamasov, Ray Soucy, Nikolay Krasnoyarski, Jason Hendry, Kevin Blackham, kouak, upa, Logan Attwood, Panagiotis Moustafellos, Thomas Courbon, and Ildar Ibragimov (hope I didn't forget anyone).
The original packages.vyos.net server is still having IO performance problems and won't handle a traffic spike associated with release well. We've setup the https://downloads.vyos.io server on our new host specially for release images and will later migrate the rest of the old server including package repositories and the rsync setup.
Now that 1.1.8 release candidate is out and is (hopefully) being tested by community members, we can get back to building the future of VyOS.
It's been obvious that Quagga needs a replacement, and, since we've been using a Quagga fork inherited from Vyatta Core that includes features that never made it to the mainline quagga, even more so. The mainline Quagga still doesn't have usable commands for configuring multiple routing tables, nor they seem to actively accept patches that would be OS-specific.
The options were discussed many times and so far it seems FreeRangeRouting is the best option. It's a fork of Quagga that is being actively developed, actively accepts contributions, and already includes a number of features that Quagga lacks, such as support for network namespaces, PIM-SM, working IS-IS and more. There's also work being done on non-disruptive config reload.
While FRR is more or less a drop-in replacement for Quagga, it's not identical, and many CLI adjustments will be needed to make VyOS work with it. It needs a lot of testing. For this I've built a custom image that has vyatta-quagga replaced with FRR.
You can download the image here: http://dev.packages.vyos.net/tmp/vyos-1.2.0-alpha-frr-test.iso Please test it and report any issues in the routing protocols configuration you find. It's obviously experimental and you shouldn't use it in real routers, the best way to test is to load your production configs into test virtual machines.
Now to the GPL violation we've found. That violation in fact has been there for over five years and no one noticed it! Then again, it's relatively indirect and subtle.
Quagga is licensed under GPL (and so is FRR). In Vyatta/VyOS, Quagga has been built with SNMP support, so it links with net-snmp. In turn, net-snmp is built with SSL support and links with OpenSSL. This is where the problem is, OpenSSL is licensed under a four-clause BSD license that is not compatible with GPL.
Sadly, there is no easy way out, so it will take some time to fix this violation. The options are:
The third option is hardest to implement, but it's the most appealing of all since all functionality will be preserved. We'd like to hear your suggestions regarding the libraries that would be license compatible and that can co-exist with OpenSSL.
A new image is available for testing: http://dev.packages.vyos.net/iso/testing/vyos-1.1.8-rc2-amd64.iso
If you are using VLAN interfaces, avoid the rc1 one and install the rc2 right away. Thanks to Samir we have identified a bug in VLAN interface configuration scripts (https://phabricator.vyos.net/T435) that prevented them from loading correctly. It was caused by an overlooked syntax error in the new options for ingress/egress VLAN QoS options.
Oddly, the current branch had a correct version of that patch, so 1.2.0 builds are not affected.
The long overdue 1.1.8 release candidate is available for download from http://dev.packages.vyos.net/iso/testing/vyos-1.1.8-rc1-amd64.iso
While a number of people have already been running 1.2.0 nightly builds in production, we do acknowledge there are people who are not in position to install updates that are not completely stable, and recently discovered vulnerabilities in dnsmasq that potentially allow remote code execution are impossible to ignore (unlike many older vulnerabilities that are only locally or aren't practical to exploint).
It's stable for all practical purposes, but since it includes pretty big updates and a few new features, I suppose it's better to go through the release candidate phase. If in say a week no one finds any issues.
The release is only available for 64-bit machines at the moment. We can provide it for 32-bit, but we are wondering if anyone still wants it, when even small boards have 64-bit CPUs.
You can read the full changelog here: https://wiki.vyos.net/wiki/1.1.8_changelog_proposal
Among package updates, there are openssl 1.0.2l and dnsmasq 2.72. Since squeeze is long EOL, the OpenSSL update required re-compiling everything that depends on OpenSSL ourselves, which took longer than we hoped.
Among VyOS fixes and features, there are user/password authentication for OpenVPN, as-override option for BGP neighbors, as-path-exclude option for route-map rules, tweakable pipe (buffer) size for netflow/sflow (too small hardcoded value could cause pmacct crash on high traffic routers), peer-to-peer VXLAN interfaces, and multiple fixes for bugs of varying severity, such as overly high CPU load on KVM guests or protocol negation in NAT rules not working.
A lot of features from 1.2.0 are not backportable due to big code changes and dependencies on way newer software versions than 1.1.x could provide, so features for cherry-picking had to be carefully chosen and even that needed quite a bit of merge conflict resolution. Quite a few of those were meant for the ill-fated "lithium" release that was supposed to be named 1.2.0 and be the last squeeze-based release, but then squeeze EOL'd, then serious life circumstances forced Alex Harpin to put all his VyOS work on hold thus leaving the maintainers team even more understaffed, and then the company we started to fund VyOS development through commercial support and services had a hard time when it almost reached the point of bankruptcy and dissolution (and, since it's self-funded, its founders almost reached the point of personal bankruptcy along with it), so by the time we could get things back on track a feature release based on squeeze wouldn't be feasible, especially considering how much we had to change to make the old codebase run on jessie. In a sense, it's a lithium that could have been, at least partially, rather than a straight maintenance release with nothing but bugfixes.
But, many of those features spent so much time in the limbo without making it into a release called stable that we felt compelled to include at least some of them.
I would like to say thanks to everyone who contributed and made this release possible, namely: Kim Hagen, Alex Harpin, Yuya Kusakabe, Yuriy Andamasov, Ray Soucy, Nikolay Krasnoyarski, Jason Hendry, Kevin Blackham, kouak, upa, Logan Attwood, Panagiotis Moustafellos, Thomas Courbon, and Ildar Ibragimov (hope I didn't forget anyone).
Our community member Markus Stubbig wrote a VyOS book in German that is now available for purchase from Amazon or bob.de.
Markus says there are no definite plans for an English version yet because he's not confident about his translation skills and will need help. If you have those skills and want to offer your services, we can connect you with Markus.
We don't have copies of the book yet (and none of the VyOS maintainers are fluent in German either, though I can read it a little), we cannot provide any kind review of the book yet, but we have no reasons to doubt Markus' expertise. If you get the book and write a review, we may publish it in the blog.
We have fixed the issue with key fetching and submitted the updated AMI for review. It passed the automated scan, but manual review and deployment to the marketplace will take some time.
The new AMI also includes updates for dnsmasq security vulnerabilities that will be included in 1.1.8. If you want to install those updates on 1.1.7 by hand, you can use these packages: http://dev.packages.vyos.net/tmp/dnsmasq/
Quick facts: the issue is caused by an unexpected change in the EC2 system, there is no solution or workaround yet but we are working on it.
In the last week a number of people reported an issue with newly created EC2 instances of VyOS where they could not login to their newly created instance. At first we thought it may be an intermittent fault in the AWS since the AMI has not changes and we could not reproduce the problem ourselves, but the number of reports grew quickly, and our own test instances started showing the problem as well.
Since EC2 instances don't provide any console access, it took us a bit of time to debug. By juggling EBS volumes we finally managed to boot an affected instance with an disk image modified to include our own SSH keys.
The root cause is in our script that checks if the machine is running in EC2. We wanted to produce the AMI from an unmodified image, which required inclusion of the script that checks if the environment is EC2. Executing a script that obtains an SSH key from a remote (even if link-local) address is a security risk since in a less controlled environment an attacker could setup a server that could inject their keys to all VyOS systems.
The key observation was that in EC2, both system-uuid and system-serial-number fields in the DMI data always start with "EC2". We thought this is a good enough condition, and for the few years we've been providing AMIs, it indeed was.
However, Amazon changed it without warning and now the system-uuid may not start with EC2 (serial numbers still do), and VyOS instances stopped executing their key fetching script.
We are working on the 1.1.8 release now, but it will go through an RC phase, while the solution to the AWS issue is needed right now. We'll contact Amazon support to see what are the options, stay tuned.