VyOS 2.0 development digest #8: vote for or against the new tag node syntax, and the protobuf schema

Tag node syntax

The change in tag node format I introduced in the previous post turned out quite polarizing and started quite some discussion in the comments. I created a poll in phabricator for it: https://phabricator.vyos.net/V3 , please cast your vote there.

If you missed the post, or found the explanation confusing, here's what it's all about. Right now in config files we format tag nodes (i.e. nodes that can have children without predefined names, such as interfaces and firewall rules) differently from other nodes:


/* normal node */
interfaces {
  /* tag node */
  ethernet eth0 {
    address 192.0.2.1/24
  }
  /* tag node */
  ethernet eth1 {
    address 203.0.113.1/24
  }
}

It looks nice, but complicates the parser. What I proposed and implemented in the initial parser draft is to not use any custom formatting for tag nodes:

/* normal node */
interfaces {
  /* actually a tag node, but rendering is now the same as for normal */
  ethernet {
    eth0 {
      address 192.0.2.1/24;
    }
    eth1 {
     address 203.0.113.1/24
    }
  }
}

This makes the parser noticeable simpler, but makes the syntax more verbose and adds more newlines.

If more people vote against this change than for it, I'll take time to implement it in the parser.

Note: This change only affects the config syntax, and has no effect on the command syntax. The command for the example above would still be "set interfaces ethernet eth0 address 192.0.2.1/24", in user input and in the output of "show configuration commands". Tag nodes will also be usable as edit levels regardless of the config file syntax, as in "edit interfaces tunnel; copy tun0 to tun1".

Protobuf schema

Today I wrote an initial draft of the protobuf schema that VyConf daemon will use for communication with clients (shell, CLI tool, and HTTP bridge). You can find it here: https://github.com/vyos/vyconf/blob/master/data/vyconf.proto

Right now it defines the following operations:

SetupSession

Setup the VyConf communication session. Takes two optional arguments: ClientApplication and OnBehalfOf. ClientApplication is purely informational, it will show up in the commit log and other accounting logs just like we see "root by boot-config-loader" or "vyos by cli" in "run show system commit" now.

OnBehalfOf is more interesting, it's meant for HTTP bridge and possibly other API bridges (netconf etc.) that use their own authentication mechanism. The interactive shell and the CLI client when run locally will use SO_PEERCRED for authentication, because no one will ever want to enter their password in the VyOS shell after entering it to login via SSH, but the only alternative to SO_PEERCRED is full-fledged single sign-on through Kerberos. Since the HTTP bridge is a separate process that takes requests from non-local users, VyConf cannot take the UID from socket credentials, and needs another way to get it. Obviously, only special processes (e.g. that run with vyconfapi GID) will be allowed to set OnBehalfOf.

EnterConfigurationMode

That's what the shell will use for the "configure" command. Takes two optional arguments: Exclusive and OverrideExclusive. Exclusive mode will be an advisory lock, to avoid having to kill sessions in situations when a user left session open and went home, or a transition script hung. In the interactive shell, configure will display a message like:

$ configure
Warning: configuration is locked, user dmbaturin entered exclusive mode on 2017 Jan 06 22:20
If you are certain that you will not interfere with their work, you can override the lock.
Enter configuration mode? [n/Y]

Set, Delete

These are obvious I guess. They take configuration path. Set optionally takes argument "ephemeral" that makes the node market ephemeral, which means it will not appear in the saved config (this is for HA transition scripts, IDS tools and similar applications).

Copy, Rename

These take the base path, source, and destination. This means, among other things, that there will be no "copy rule 10 to rule 20" for the user, just "edit ... rule; copy 10 to 20".

Commit, Rollback

Commit takes optional Confirmed and ConfirmTimeout arguments for commit-confirm, and also optional Comment arguments for adding commit descriptions. Rollback takes Revision argument.

Comment

Adds a comment to a node.

ShowConfig

Takes optional Path argument for displaying parts of the config, and also optional Format argument that right now can be Curly or JSON. Instead of reading individual values through return_value(), config scripts (and web GUI) will be able to use the JSON output of ShowConfig to get the complete configuration of a node in machine-readable format, it can also be useful for external tools that work with the config (including migration scripts).

Exists, GetValue, GetValues, ListChildren

These will hardly be used by interactive tools, they are meant for config scripts, just like the exists(), returnValue(), returnValues(), and listNodes() functions we have now.

RunOpMode

Takes a path, runs an op mode command.

What's next

Next these operations will be wrapped into a higher level client library. At the same time I'll add some message routing, and then we will be able to start VyConf and send some commands to it through a non-interactive client, and have some fun with it. A runnable application will make it easier to test the features we are going to develop next, such as config rendering, building and displaying config diffs, and running actual config scripts at commit time.

8 responses
Daniil, it's such a gift to have you as VyOS system architect and developer
Robert, thanks for the kind words! I really hope these changes will finally make VyOS easy enough to contribute and contributors will start coming. If this will not, then VyOS is officially dead.
Hi All. For device config management is easier nowadays use boot config file than set commands. You can generate whole config and you know thanks to checker during load that is valid. Managing device through set commands is tricky at least when it comes to deletion/changes. In future would be great to have config file that will be always backward compatible, or other method of configuration eg Json in some format that will not change quickly maybe even e.g. Open config. Thanks for great project ! :)
I'll add a vote to implement the combined tag nodes in the parser, I'm coming around with the ; as an endpoint rather than a newline as it may give the option to add comments post semicolon? Also combining the interface and its descriptor allows easier matching against competitor (proprietary) solutions.
Alex, if you mean pesistent comments that can be set with "comment" command and are enclosed in /* */, then I'm afraid there is no way to both adhere to the principle of least surprise and be consistent. Suppose we make it so that comments are after the node, then things like firewall rulesets may have their comment hundreds lines below the node name. If we make it so that leaf nodes have their comments after the node but all other nodes have it before, then it's a massive gotcha for anyone who wants to edit configs by hand. Non-persistent comments that are ignored by the parser can be places anywhere, however. Piotr, sorry for late reply, but I'm seriously not sure what's so handy about editing configs by hand. Generating structures syntax is not quite trivial even if doable, and then I have no idea what you are talking about when you say deletion with set commands is tricky. We went to great length in Vyatta/VyOS 1.x to make sure it's convenient, safe, and observable through diffs, and even more work awaits in 2.0. There is also completion that allows you to type it even if you don't remember the syntax offhand, something config editing will never give you (unless you import all interface definitions and write and editor plugin to autocomplete config keywords), and you get to keep those definitions up to date on your desktop too. As of backwards compatibility, well, there will be none. It's only possible with a MASSIVE safety compromise, since you get to allow and ignore unknown options, and then you cannot tell if it's a new option from a future version, or a typo. Forward compatibility is another story of course, we have always taken it seriously and always will, I hope 2.0 release will be the only forward-incompatible relase in a loong while.
3 visitors upvoted this post.