OSTree: Between packages and images
Why am I here
- Free Software
- Fun
- Red Hat Enterprise Linux
Fedora/Debian: Package + Install Image software delivery
- All delivery and infrastructure is structured in terms of packages
- Install images: Work once, but is decomposed into package set
thereafter for updates
- Comps: Until YumGroupsAsObjects, also was decomposed
- "Critical path"
ChromeOS: Delivered as unit, with atomic upgrades
- Targets specific hardware
- Supports apps and extensions
- Problem: local development tools
Red Hat Enterprise Linux (Server|Workstation)
- And now Fedora.next products
- Need a model where we can meaningfully have both a Workstation
and a Server
- Fedora cannot just be a desktop we retrofit for the server
(DHCP, footprint...)
OSTree background
- Part of research project to improve GNOME -> Red Hat Enterprise
Linux flow and quality
- Developed in concert with gnome-continuous (now very fast CI/CD)
- But intentionally separate to support and augment dpkg/rpm
Set of packages <-> tree
- You don't really "run Debian/Fedora" - you run a bootable
filesystem tree (kernel+/usr) resulting from installation of
a subset of packages
- OSTree requires naming these trees: @standard docker -> fedostree/20/x86_64/server/docker
- You can replicate precomputed trees from from a build server
- Trees are versioned - you can upgrade (plain HTTP)
- ostree admin upgrade - a fully atomic process
More flexibility than ChromeOS
- You can switch between trees at any time, and keep your
changes to /etc and /var!
- ostree admin switch fedostree/rawhide/x86_64/server/docker
- ostree admin switch fedostree/20/x86_64/workstation/gnome/core
- You can go backwards:
- ostree admin deploy 4b2b89bb...
Implementation: OSTree layout
- Boot into chroot in /ostree/deploy/fedostree/deploy/checksum
- Content of /usr is hardlinked to /ostree/repo
- Dracut (initramfs) hook which makes deployment chroot be /
- Atomic swap of /boot/loader config files
- Works on top of any Unix filesystem/block storage
Implementation: Upgrades
- Fetch new ref, returns checksum
- Download metadata and objects we don't have, each one HTTP request
- Create new chroot
- Checkout /etc, apply diff of old default versus current on top
- Atomic swap of /boot/loader which is symlinked to /boot/loader.[01]
Implementation: Changes to OS content
- /usr/etc contains default contents of /etc - 3 way merge
- /usr/lib/passwd contains system users
- Files in /var -> tmpfiles.d snippets
Phase 0: rpm-ostree build/integration server in Fedora
- Alongside packges
- Commit trees for Fedora.next products to OSTree repository (GPG, etc.)
- Push notification from koji
- Test new trees using virtualization
- Client systems can replicate them - safely test rawhide
Phase 1: Project becomes tree aware
Phase 2: Extended deployment
- Integration with Docker (and other "application mechanisms")
- Compose packages on client side
- Continue experimentation with client app frameworks (only for FOSS)
Phase 3: non-replication software management and deployment
- OSTree based "layer composition" (layer = group of packages)
- fedostree/20/x86_64/gnome/workstation/core+firefox+@gnome-games
- No removals of individual packages - can only compose from lower level
- pkglayer install strace works everywhere
- yum install @minimal firefox produces nonsensical result
- pkglayer install firefox outputs error that it requires "uishell"
- No tool to uninstall pulseaudio or whatever; must use systemctl mask
Phase 3: Live update application
- Research applying updates live without race conditions where possible
- Runtime-only overlay: Stop service, bind mount new files over
top of old ones, start service
Testing
- Packaging is necessary
- ...but not sufficient
- rpmlint is not going to find memory leaks
Testing trees very, very quickly
- Time-To-Screenshot metric
- The "totally offline" model of OSTree updates allows
incrementally updating a cached VM disk image from the
host via libguestfs
- Ability to reboot into previous tree very powerful for Beaker type physical
hardware testing
Trying it out