blog.vyos.net

Ok, I changed my mind: before we jump into the abyss of datastructures and look how the config and reference trees work, I'll describe the changes I made in the last few days.

Also, I'd like to make it clear that if you don't respond to design questions I state in these posts, I, or whoever takes up the task, will just do what they think is right. ;)

I guess I'll start with the questions this time. First, in the comments to the first post, the person who goes by kglkgvkvsd544 suggested two features: commit-confirm by default, and an alternative solution to the partial commit where instead of loading the failsafe config, the system loads a previous revision instead, in the same way as commit-confirm does. I don't think any of those should be the only available option, but having them as configurable options may be a good idea. Let me know what you think about it.

Another very important question: we need to decide on the wire protocol that vyconf daemon will use for communication with its clients (the CLI tool, the interactive shell, and the HTTP bridge). I created a task for it, https://phabricator.vyos.net/T216, let me know what you think about it. I'm inclined towards protobuf myself.

Now to the recent changes.

vyconfd config

As I already said, VyConf is supposed to run as a daemon and keep the running config tree, the reference tree, and the user sessions in memory. Obviously, it needs to know where to get that data. It also needs to know where to write logs, where the PID file and socket file should be, and other things daemons are supposed to know. Here's what the vyconf config for VyOS may look like:

[appliance]
name = "VyOS"

data_dir = "/usr/share/vyos/"
program_dir = "/usr/libexec/vyos"
config_dir = "/etc/vyos"

# paths relative to config_dir
primary_config = "config.boot"
fallback_config = "config.failsafe"

[vyconf]
socket = "/var/run/vyconfd.sock"
pid_file = "/var/run/vyconfd.pid"
log_file = "/var/log/vyconfd.log"

That INI-like language is called TOML, it's pretty well specified and capable of some fancy stuff like arrays and hashes, apart from simple (string, string) pairs. The best part is that there are libraries for parsing it for many languages, and the one for OCaml is particularly nice and idiomatic (it uses lenses and option types to access a nested and possibly underdefined datastructure in a handy and typesafe way), like:

let pid_file = TomlLenses.(get conf (key "vyconf" |-- table |-- key "pid_file" |-- string)) in
match pid_file with
| Some f -> f
| None -> "/var/run/vyconfd.pid"

The config format and this example reflects some decisions. First, the directory structure if more FHS-friendly. What do we need /opt/vyos for, if FHS already has directories meant for exactly what we need: architecture-independent data (/usr/share), programs called by other programs and not directly by users (/usr/libexec), and config files (/etc)?
Second, all important parameters are configurable. The primary motivation for this is making VyConf usable for every appliance developer (most appliances will not be called VyOS for certain), but for ourselves and for every contributor to VyOS it's a reminder to avoid hardcoded paths anywhere: if changing it is just one line edit away, a hardcoded path is an especially bad idea.

Here's the complete directory structure I envision:

$data_dir/
  interfaces/ # interface definitions
  components/ # Component definitions
$program_dir/
  components/  # Scripts/programs that verify the appliance config, generate actual configs, and apply them
  migration/       # Migration scripts (convert appliance config if syntax changes)
  validators/       # Value validators
$config_dir/
  scripts/              # User scripts
  post-config.d/   # Post-config hooks, like vyatta-postconfig-bootup.script
  pre-commit.d/   # Pre-commit hooks
  post-commit.d/ # Post-commit hooks
  archive/             # Commit archive

This is not an exhaustive list of directories an appliance can have of course, it's just directories that have any significance for VyConf. I'm also wondering if we should introduce post-save hooks for those who want to do something beyond the built-in commit archive functionality.

External validators

As you remember, my idea is to get rid of the inflexible system of built-in "types" and make regex the only built-in constraint type, and use external validators for everything else.

External validators will be stored in the $program_dir/validators. Since many packages use the same types of values (IP addresses is a common example), and in VyOS 1.x we already have quite a lot of templates that reference the same validation scripts, making it a separate entity will simplify reusing them, since it's easy to see what validators exist, and you can also be sure that they behave like you expect (or if they don't, it's a bug).
Validator executable take two arguments, first argument is constraint string, and the second is the value to be validated (e.g. range "1-99" "42").The expected behaviour is to return 0 if the value is valid and a non-zero exit code otherwise. Validators should not produce any output, instead the user will get the message defined in the constraintError tag of the interface definition (this approach is more flexible since different things may want to use different messages, e.g. to clarify why exactly the value is not valid).

That's all for today. Feel free to comment and ask questions. The next post really will be about the config tree and the way set commands work, stay tuned.