Building a Kubernetes-ready Fedora CoreOS image
Published on Jun 06
Getting our environment ready
In order to build our Kubernetes-ready Fedora CoreOS image, we’ll need a couple of things. First of all, we need a base image - a Fedora CoreOS ISO. Once we have that, we need to install Butane in order to create our configuration.
Fedora CoreOS ISO
Fedora CoreOS will be the base of this project, it’s an open-source operating system and part of the Fedora Project (which is a community effort sponsored by Red Hat themselves). The OS itself is designed for running containerized workloads and cloud-native applications, which makes it perfect for running a Kubernetes cluster.
Some of its most noteworthy (slightly opinionated) features are:
- Container-First Approach: Fedora CoreOS is specifically designed to run containers as the primary workload. It provides a minimal, immutable base operating system that focuses on running containerized applications efficiently.
- Automatic Updates: Fedora CoreOS has a system that handles automatic updates, which updates the whole operating system at once. This approach improves security and simplifies maintenance by providing seamless, reliable updates without requiring manual intervention.
- Immutable Infrastructure: In combination with Ignition, a Fedora Coreos image can be prepared with a specific configuration. Using this prepared image, we can create multiple machines with the exact configuration.
- Minimalist and Modular: The operating system is minimalistic by design, including only essential components needed to run containers. However, it offers modularity, allowing additional components and services to be installed as needed.
- Open-Source and Community-Driven: Fedora CoreOS is an open-source project, built by the community and sponsored by Red Hat. It benefits from the collaboration and contributions of developers and users who work together to improve and enhance the operating system.
The latest Fedora CoreOS iso can be downloaded from their site.
Butane and Ignition
Butane is a YAML-based configuration language used for provisioning Fedora CoreOS (and Red Hat CoreOS) images. It simplifies the process of creating and managing Ignition configurations. Ignition is the first process to run on a CoreOS machine and applies the desired system configuration.
Using Butane, we can preconfigure things such as disk partitioning, file system creation, user creation, network configuration, and more. All of this can be done in a declarative YAML format, making it easier to manage and version control compared to Ignition.
To use Butane, we first need to install the CLI tool. This can be done using DNF on Fedora machines: dnf install -y butane. If you’re not running any Fedora(-based) OS, you can also download the binary from the Github releases page. After that, you can write a Butane configuration file that describes the desired system state, including the various configuration parameters. You can then pass this file to the Butane utility, which converts it into an Ignition configuration that can be applied during the system boot process.
Once you have your Butane configuration file ready, you can pass it to the butane command-line utility to generate the corresponding Ignition configuration: butane -o ignition.json butane-config.yaml.
The resulting ignition.json file can then be supplied to the CoreOS installation process, which will apply the configuration during the initial system boot.
Preparing a Fedora CoreOS image
It’s time to write a Butane configuration file, yay! For this image we want a couple of things to be done:
- Create a user which we can use to log onto the machine
- Add the CRI-O DNF module
- Add the Kubernetes YUM repository
- Prepare networking for Kubernetes
Translating these requirements into a Butane configuration file, we get something like this:
variant: fcos
version: 1.0.0
passwd:
users:
- name: robot
password_hash: "" # Add your hashed password (using mkpasswd from the whois package) here!
ssh_authorized_keys:
- "ssh-rsa " # Add your SSH key here!
groups: [ sudo, docker ]
storage:
directories:
- path: /var/cache/rpm-ostree-install
files:
# CRI-O DNF module
- path: /etc/dnf/modules.d/cri-o.module
mode: 0644
overwrite: true
contents:
inline: |
[cri-o]
name=cri-o
stream=1.27
profiles=
state=enabled
# YUM repository for kubeadm, kubelet and kubectl
- path: /etc/yum.repos.d/kubernetes.repo
mode: 0644
overwrite: true
contents:
inline: |
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
# configuring automatic loading of br_netfilter on startup
- path: /etc/modules-load.d/br_netfilter.conf
mode: 0644
overwrite: true
contents:
inline: br_netfilter
# setting kernel parameters required by kubelet
- path: /etc/sysctl.d/kubernetes.conf
mode: 0644
overwrite: true
contents:
inline: |
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
systemd:
units:
- name: rpm-ostree-install@.service
enabled: true
contents: |
[Unit]
Description=Layer %i with rpm-ostree
Wants=network-online.target
After=network-online.target
Before=zincati.service
ConditionPathExists=!/var/cache/rpm-ostree-install/%i.stamp
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/flock /var/cache/rpm-ostree-install/.lock -c "/usr/bin/rpm-ostree install --assumeyes --idempotent --reboot --allow-inactive %i | tee /var/cache/rpm-ostree-install/%i.stamp"
[Install]
WantedBy=multi-user.target
- name: rpm-ostree-install@python3.service
enabled: true
- name: rpm-ostree-install@libselinux-python3.service
enabled: true
Now that we’ve created our Butane configuration, it’s time to convert it to an Ignition file. We can do that by executing the following command: butane fcos.bu -o fcos.ign. This command now generates an ignition file which will look something like this:
{"ignition":{"version":"3.0.0"},"passwd":{"users":[{"groups":["sudo","docker"],"name":"robot","passwordHash":"","sshAuthorizedKeys":["ssh-rsa "]}]},"storage":{"directories":[{"path":"/var/cache/rpm-ostree-install"}],"files":[{"overwrite":true,"path":"/etc/dnf/modules.d/cri-o.module","contents":{"source":"data:,%5Bcri-o%5D%0Aname%3Dcri-o%0Astream%3D1.27%0Aprofiles%3D%0Astate%3Denabled%0A"},"mode":420},{"overwrite":true,"path":"/etc/yum.repos.d/kubernetes.repo","contents":{"source":"data:,%5Bkubernetes%5D%0Aname%3DKubernetes%0Abaseurl%3Dhttps%3A%2F%2Fpackages.cloud.google.com%2Fyum%2Frepos%2Fkubernetes-el7-x86_64%0Aenabled%3D1%0Agpgcheck%3D1%0Arepo_gpgcheck%3D1%0Agpgkey%3Dhttps%3A%2F%2Fpackages.cloud.google.com%2Fyum%2Fdoc%2Fyum-key.gpg%0A%20%20https%3A%2F%2Fpackages.cloud.google.com%2Fyum%2Fdoc%2Frpm-package-key.gpg%0A"},"mode":420},{"overwrite":true,"path":"/etc/modules-load.d/br_netfilter.conf","contents":{"source":"data:,br_netfilter"},"mode":420},{"overwrite":true,"path":"/etc/sysctl.d/kubernetes.conf","contents":{"source":"data:,net.bridge.bridge-nf-call-iptables%3D1%0Anet.ipv4.ip_forward%3D1%0A"},"mode":420}]},"systemd":{"units":[{"contents":"[Unit]\nDescription=Layer %i with rpm-ostree\nWants=network-online.target\nAfter=network-online.target\nBefore=zincati.service\nConditionPathExists=!/var/cache/rpm-ostree-install/%i.stamp\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=/usr/bin/flock /var/cache/rpm-ostree-install/.lock -c \"/usr/bin/rpm-ostree install --assumeyes --idempotent --reboot --allow-inactive %i | tee /var/cache/rpm-ostree-install/%i.stamp\"\n\n[Install]\nWantedBy=multi-user.target\n","enabled":true,"name":"rpm-ostree-install@.service"},{"enabled":true,"name":"rpm-ostree-install@python3.service"},{"enabled":true,"name":"rpm-ostree-install@libselinux-python3.service"}]}}
As we can see, this file contains the same data as the Butane file, but has a different syntax. This is what I was talking about in “Butane and Ignition”, Butane has a much more readable format which allows for better control.
Creating an ISO file
The first step in this process is to install the CoreOS Installer. You can obtain the latest version of the binary from the project’s Github release page. If you’re running a Fedora(-based) operating system, you can also use the dnf install coreos-installer command. One other option for installing the CLI tool is by using the Cargo package manager; cargo install coreos-installer.
With the CoreOS Installer installed and the Ignition configuration file ready, it’s time to package the configuration along with the Fedora CoreOS ISO into a new image. This ISO image will server as a self-contained package that automatically configures the system it’s installed on. This can be done by executing the following command:
coreos-installer iso customize \
--dest-device /dev/sda \
--dest-ignition fcos.ign \ # Replace this value with the Ignition file
--dest-console ttyS0,115200n8 \
--dest-console tty0 \
-o fcos-37-*$*(date +"%Y%m%d").iso \
./fedora-coreos-38.20230819.3.0-live.x86_64.iso # Replace this with the ISO you downloaded
Once the command has completed, you’ll see an ISO image with the current date in your working directory. You can now boot off this image and get a preconfigured machine. One thing you should note; the machine might reboot a couple of times while it’s configuring. Just boot the machine and go grab a coffee, after 2-3 boots it should be ready to be used.
Conclusion
Building a Kubernetes-ready Fedora CoreOS image streamlines the process of setting up a Kubernetes cluster. By following the steps outlined in this post, developers and administrators (or just DevOps engineers) can create resilient infrastructure for running Kubernetes workloads. Working with Fedora CoreOS to run Kubernetes simplifies management, provides automatic updates, and enhances security. In a next post we’ll set up a Kubernetes using the image we created.