Devconf.cz 2018

Colin Walters, Red Hat, Inc.

keybase.io/walters | walters@{redhat.com,verbum.org}

Types of software

• Host: kernel, systemd, lvm, PAM, sshd
• Extensions: VPNs, pcscd+ykclient, tmux, powerline; and fonts, codecs
• Host or (system) container: docker, kubernetes
• Tracing/debugging: perf, systemtap
• Development: gcc, foo-devel
• Misc cmdline: git, ansible, gpg2, fdisk
• Unprivileged apps (desktop + server)

Delivery mechanisms

• OS image
• App framework
• Packages (deb/RPM etc.)

Delivery mechanisms (blurred)

• Packages in a privileged container (tracing/debugging, misc CLI)
• Privileged applications
• Packages: Possible but not good for all types
• OS image + Explicit extension "slots"
• Custom built OS image

For example: Wireguard VPN

For desktops, may want GUI config tool. And across server/desktop, want headless backend. Currently e.g. flatpak is "pure desktop app" only and doesn't have a "host dependency" mechanism.

Except systemd-networkd now supports it

OS image + apps

Android, iOS, CoreOS, Ubuntu Core, ChromeOS, Resin OS

And of course Atomic Host!

Most have different app/OS frameworks

Most have "blurred" OS/app/extension mechanisms
(e.g. iOS VPN)

ChromeOS - extensions

arstechnica: "Malicious Chrome extension is next to impossible to manually remove"

Ubuntu Core - snappy

debs are just one way to build snaps

"classic" confinement = full access to the system

Kubernetes DaemonSet

Run a container on each node

Often privileged, primary examples are
storage agents, monitoring

Compromise of registry = compromise of cluster
Maybe require GPG signatures for these?

Atomic system containers

OS image + app framework

Like peanut better and jelly

Focus on OS image

Atomic, online (but background) updates + rollback = fearless updates with low downtime

OS image implementations

Dual partition: ChromeOS, mender.io

Resin.io/Balena (uses Docker/OCI for host)

Other: OLPC updater, Conary, Clear Linux, ...

My project: OSTree

More related projects

Also comparisons with e.g. Snapper

(lib)ostree

Gets content from A to B, GPG signatures

Works on filesystem level, not block level
i.e. anywhere rpm/dpkg work, ostree works

Deduplicating object store in /ostree/repo, hardlinks

Implements transactional updates, handles bootloader

C shared library

Flexible (e.g. EndlessOS, QtOTA, flatpak), also content-agnostic

Writing a new package manager? link to libostree!

Atomic Host base: rpm-ostree

True hybrid image + package system

BTW, ostree = libostree now

libdnf (C) provides RPM functionality (parts used by dnf)

Combination: Lots of radical changes, but leverages existing
(perhaps a bit boring) RPM ecosystem

"hybrid": like gas+electric cars

Images = electric, Packaging = gas (obviously!)

Hybrid ideal: Best of both worlds

So rpm-ostree supports "OS Extensions" (RPMs)
e.g. PAM modules

Also most existing RPMs (apps, profiling/debugging, ...)

(And maybe enable system container + RPM)

rpm-ostree features

• Atomic upgrades
• Background (cancellable!) upgrades
• Always read-only /usr
• Reliable rollback (your data in /var)
• Not opinionated about config mgmt: /etc works; use kickstart/ansible/puppet...
• Client-side: package layering/override
• ostree: static deltas (better than deltarpm)
• Easily build your own server-side

The basics

Server: rpm-ostree compose = list of RPMs → OSTree commit

fedora-atomic & fedora-atomic-ws

Dependency resolution, %post scripts (ldconfig), SELinux labeling all server side; client side bit-for-bit reproducibility

Client side view

OSTree commit has GPG sig, version number, timestamp, checksum

Upgrade pulls from "remote" + branch (like git)

# rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora/27/x86_64/atomic-host
                   Version: 27.44 (2018-01-01 21:54:31)
                    Commit: b5845ebd002b2ec829c937d68645400aa163e7265936b3e91734c6f33a510473
              GPGSignature: Valid signature by 860E19B0AFA800A1751881A6F55E7430F5282EE4
# rpm-ostree upgrade
...
... Version: 27.61 (2018-01-17 15:52:47)
                    

Originally, "rpm-" part was just pkg diffs

Fearless rebases

# rpm-ostree rebase fedora-atomic-27:fedora/27/x86_64/atomic-host
...
# rpm-ostree status
  fedora-atomic-27:fedora/27/x86_64/atomic-host
                   Version: 27.44 (2018-01-01 21:54:31)
                    Commit: b5845ebd002b2ec829c937d68645400aa163e7265936b3e91734c6f33a510473

● fedora-atomic:fedora/26/x86_64/atomic-host
                   Version: 26.141 (2017-10-05 09:29:15)
                    Commit: 541abd650d1ffb3929e2ba8114436a0b04ee41da76a691af669dd037589a1421
                    

Also if you want real excitement, rpm-ostree ex livefs --replace

Tap existing RPM ecosystem

# rpm-ostree install libvirt
...  (Maybe other system prep, followed by reboot)
# rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora/27/x86_64/atomic-host
                   Version: 27.44 (2018-01-01 21:54:31)
                BaseCommit: b5845ebd002b2ec829c937d68645400aa163e7265936b3e91734c6f33a510473
           LayeredPackages: libvirt
#
                    

Hybrid engaged, upgrade pulls from both (ostree remote + yum.repos.d)

Package installation engine: 🆕

Every operation checks out base tree, packages "reinstalled" (hardlinks) then scripts are run in a container with /var read-only, and the base /etc defaults.

Overrides

# rpm-ostree override remove docker
...
# rpm-ostree status
...
     RemovedBasePackages: docker-2:1.13.1-44.git584d391.fc27.x86_64
# curl -L -O http://myrepo.example.com/iptables-patched.x86_64.rpm
# rpm-ostree override replace ./iptables-patched.x86_64.rpm
                    

NB: Explicitly distinct from layering

Back to base image

# rpm-ostree override reset --all
# rpm-ostree uninstall libvirt # (...and other layered packages)
                    

Hybrid disengaged, back to pure ostree

Other hybrid bits

# ostree show --raw fedora-atomic:fedora/27/x86_64/atomic-host
  ... <'Fedora-27-updates...'>, 'timestamp': <uint64 1516203727>
#
                    

rpm-ostree injects OSTree metadata with rpm-md info

Down the line: Add versions to rpm-md?

Project status

• Upstream releases ~monthly (single stream)
• (Contrast update_engine, mender.io, snapper - boring git logs)
• Stable
• Supported on Fedora/CentOS/EL family
• Headlining project: Atomic Host
• Now used in side project: Atomic Workstation

Representative issues

• Packages installing into /opt
• Live updates are experimental
• DKMS needs total overhaul for this
• Docs + familiarity

Briefly: server-side composes

• Stable, but requires learning investment
• libostree is multi-vendor upstream project, incl. embedded/IoT community (rpm-ostree is separate)
Win: the same client + server tools works for you
Win: We've tested the package set in this mode
But not (yet) a Red Hat product

What's next for rpm-ostree

#247: Really polished 🆕⌚ automatic updates

Better auto rolling reboots of Kube cluster. Obvious default for desktops.

After that

yum vs atomic vs rpm-ostree (vs flatpak)

syscontainers + RPMs