Vincent De Borger / Posts / Lab 254 - GitOps with Flux

Lab 254 - GitOps with Flux

kubernetesfluxgitopssops

In the previous post, I set up the base of my homelab cluster using Talos and Cilium. I ended that post by saying the next step would be getting Flux set up to automatically install some workloads on the cluster, so let’s pick up right where we left off. In this post I’ll go over how I set up Flux, what the repository’s folder structure looks like and what the full process to install a cluster looks like now.

So, what is Flux again?

Flux is a tool for continuous delivery in Kubernetes that embraces the GitOps principle. Instead of running kubectl apply or helm install by hand, I define everything I want running in the cluster in a Git repository and let Flux make sure the cluster state matches that repository. The repository becomes the single source of truth, everything is version controlled and, most importantly for me, I don’t have to remember which commands I ran to get things into a certain state.

I already covered the basics of Flux in an earlier post, so I’ll keep the theory short here and focus on how it all fits together in this specific setup.

The folder structure

Before diving into the setup itself, it helps to understand how the repository is laid out, since Flux leans heavily on the directory structure to figure out what to apply and in which order. Everything Flux cares about lives in the kubernetes/ directory:

kubernetes/
├── bootstrap/
│   ├── github-deploy-key.sops.yaml
│   └── kustomization.yaml
├── flux/
│   ├── config/
│   │   ├── cluster.yaml
│   │   ├── flux.yaml
│   │   └── kustomization.yaml
│   ├── repositories/
│   │   ├── helm/
│   │   │   ├── cilium.yaml
│   │   │   ├── cert-manager.yaml
│   │   │   └── ...
│   │   └── kustomization.yaml
│   ├── vars/
│   │   ├── cluster-secrets.sops.yaml
│   │   ├── cluster-settings.yaml
│   │   └── kustomization.yaml
│   ├── apps.yaml
│   └── repositories.yaml
└── apps/
    ├── default/
    │   ├── podinfo/
    │   │   ├── helmrelease.yaml
    │   │   ├── helm-values.yaml
    │   │   └── kustomization.yaml
    │   └── kustomization.yaml
    ├── kube-system/
    │   ├── cilium/
    │   ├── cert-manager/
    │   └── kustomization.yaml
    └── kustomization.yaml

There’s a bit going on here, so let me walk through it top to bottom:

  • bootstrap/ is applied exactly once, by hand, and is not tracked by Flux. It installs the Flux controllers and CRDs into the cluster and holds the deploy key Flux uses to pull this repository. Think of it as the bit that has to exist before Flux can start managing itself.
  • flux/config/ is the entrypoint. It defines the GitRepository pointing at this repo and the top-level Kustomization that tells Flux to start reconciling the rest of kubernetes/flux.
  • flux/ (the apps.yaml and repositories.yaml files) contains two more Kustomization resources that branch off into kubernetes/apps and kubernetes/flux/repositories.
  • flux/repositories/ holds all the HelmRepository sources, one file per Helm repo I pull charts from.
  • flux/vars/ contains a ConfigMap and a (SOPS-encrypted) Secret with cluster-wide variables that get substituted into the manifests. More on that later.
  • apps/ is where the actual workloads live, grouped per namespace. Each namespace has its own kustomization.yaml listing its apps, and each app is a folder with a HelmRelease, its values and a kustomization.yaml.

The nice thing about this layout is that it’s a chain: config/ points at flux/, flux/ points at apps/ and repositories/, and each namespace folder points at its apps. Adding something new almost always comes down to dropping a folder in the right place and adding one line to a parent kustomization.yaml.

Keeping secrets secret with SOPS

Since I want the repository to be the single source of truth, I also want my secrets in there. Obviously I don’t want to commit plaintext passwords and tokens to Git, so I use SOPS with age to encrypt them. Flux can decrypt these on the fly when it applies them, which means I get to keep everything in one place without leaking anything.

First, I generate an age key with age-keygen and move it to the location SOPS looks for it:

age-keygen -o age.agekey
mkdir -p ~/.config/sops/age
mv age.agekey ~/.config/sops/age/keys.txt

That gives me an age key at ~/.config/sops/age/keys.txt. The age-keygen output also prints the matching public key, which I drop into the .sops.yaml file at the root of the repository. That file tells SOPS which key to encrypt with and which fields to encrypt:

creation_rules:
  - path_regex: talos/.*\.sops\.ya?ml
    key_groups:
      - age:
          - "<insert your key here>"
  - path_regex: .*\.sops\.ya?ml
    encrypted_regex: "^(data|stringData)$"
    key_groups:
      - age:
          - "<insert your key here>"

The second rule only encrypts the data and stringData fields, so the rest of the manifest stays readable in Git, which is handy when you want to see what a secret is for without decrypting it.

Next up are the two secrets Flux needs to get going. The first is cluster-secrets.sops.yaml in kubernetes/flux/vars/, which holds any cluster-wide secret values I want to substitute into my manifests:

apiVersion: v1
kind: Secret
metadata:
  name: cluster-secrets
  namespace: flux-system
stringData:
  SOME_SECRET: SOME_VALUE

The second is github-deploy-key.sops.yaml in kubernetes/bootstrap/, which is the key Flux uses to pull the repository:

apiVersion: v1
kind: Secret
metadata:
  name: github-deploy-key
  namespace: flux-system
type: Opaque
data:
  identity: YOUR_BASE64_ENCODED_PRIVATE_KEY_USED_TO_PULL_SOURCE_CODE
  known_hosts: A_BASE64_ENCODED_KNOWN_HOSTS_ENTRY

You can generate those two base64 values with:

base64 -w 0 <private_key_path>
ssh-keyscan github.com | base64 -w 0

With the .sops.yaml file configured and both secrets created, all that’s left is to encrypt them. SOPS encrypts in place, following the rules from .sops.yaml:

sops --encrypt --in-place kubernetes/flux/vars/cluster-secrets.sops.yaml
sops --encrypt --in-place kubernetes/bootstrap/github-deploy-key.sops.yaml

From here on out, any file ending in .sops.yaml is safe to commit to Git.

Bootstrapping Flux

With secrets sorted, it’s time to actually get Flux into the cluster. Before that, one small thing to adjust: the Git repository link in kubernetes/flux/config/cluster.yaml needs to point at your own repository. This file defines the GitRepository Flux watches, along with the top-level Kustomization that kicks everything off:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: lab-254
  namespace: flux-system
spec:
  interval: 30m
  url: "https://github.com/DB-Vincent/lab254.git"
  secretRef:
    name: github-deploy-key
  ref:
    branch: "main"
  ignore: |
    # exclude all
    /*
    # include kubernetes directory
    !/kubernetes
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cluster
  namespace: flux-system
spec:
  interval: 30m
  path: ./kubernetes/flux
  prune: true
  wait: false
  sourceRef:
    kind: GitRepository
    name: lab-254
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-settings
      - kind: Secret
        name: cluster-secrets

A couple of things worth pointing out here. The decryption block is what lets Flux decrypt those SOPS-encrypted secrets I mentioned earlier, using the age key. The postBuild.substituteFrom block is a neat trick, it takes the values from the cluster-settings ConfigMap and cluster-secrets Secret and substitutes them into the manifests before applying them. That way I can define something like a timezone or a CIDR once and reference it everywhere:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-settings
  namespace: flux-system
data:
  TIMEZONE: Europe/Brussels
  CLUSTER_CIDR: "10.42.0.0/16"
  SERVICE_CIDR: "10.43.0.0/16"
  NODE_CIDR: "10.252.252.0/24"
  KUBE_VIP_ADDR: 10.252.252.199

There’s also a flux.yaml next to cluster.yaml, which points Flux at the flux-manifests OCI repository and lets it manage (and update) its own components the GitOps way. It applies a few patches to bump resource limits and tweak the number of concurrent reconciliations, but I won’t paste the whole thing here, you can find it in my repository if you’re curious.

With that pointed at the right place, committed and pushed, getting Flux into the cluster comes down to a handful of commands. I’m assuming you’ve already exported KUBECONFIG so kubectl talks to the right cluster (export KUBECONFIG=$(pwd)/talos/clusterconfig/kubeconfig).

First, install the Flux controllers and CRDs using the bootstrap/ kustomization. This also creates the flux-system namespace:

kubectl apply --server-side --kustomize kubernetes/bootstrap

Next, create a sops-age secret from the age key, so Flux can decrypt the SOPS-encrypted manifests it finds in Git:

cat ~/.config/sops/age/keys.txt | \
  kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=/dev/stdin

Then apply the GitHub deploy key so Flux is able to pull the repository:

sops --decrypt kubernetes/bootstrap/github-deploy-key.sops.yaml | kubectl apply --server-side --filename -

Now the cluster variables, the cluster-secrets secret (decrypted on the way in) and the cluster-settings ConfigMap, so the postBuild substitution has something to work with:

sops --decrypt kubernetes/flux/vars/cluster-secrets.sops.yaml | kubectl apply --server-side --filename -
kubectl apply --server-side --filename kubernetes/flux/vars/cluster-settings.yaml

And finally, the moment Flux takes over. Applying the flux/config/ kustomization creates the GitRepository and the top-level Kustomization:

kubectl apply --server-side --kustomize kubernetes/flux/config

And that’s it. Once that’s done, Flux picks up apps.yaml and repositories.yaml, follows the chain through the folder structure and starts installing everything it finds. From this point on I barely touch kubectl again, if I want to change something, I change it in Git and let Flux do the rest.

Adding an application

To show how this works in practice, let’s add an application the Flux way. I’ll use podinfo since it’s a nice, simple example.

First, Flux needs to know where to pull the chart from, so I add a HelmRepository in kubernetes/flux/repositories/helm/podinfo.yaml:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 15m
  url: https://stefanprodan.github.io/podinfo

And add it to the kustomization.yaml in that folder so Flux picks it up:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cilium.yaml
  - cert-manager.yaml
  - podinfo.yaml

Next, the application itself. I create a podinfo folder in kubernetes/apps/default/ with three files. The first is the HelmRelease, which tells Flux which chart to pull and which values to use:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 30m
  chart:
    spec:
      chart: podinfo
      version: 6.7.1
      sourceRef:
        kind: HelmRepository
        name: podinfo
        namespace: flux-system
  install:
    remediation:
      retries: 3
  upgrade:
    cleanupOnFail: true
    remediation:
      retries: 3
  valuesFrom:
    - kind: ConfigMap
      name: podinfo-helm-values
      valuesKey: values.yaml

Instead of stuffing the Helm values inline, I keep them in a separate helm-values.yaml file, which keeps the HelmRelease clean and makes diffs easier to read:

replicaCount: 1

The two are tied together in the kustomization.yaml, which uses a configMapGenerator to turn helm-values.yaml into the podinfo-helm-values ConfigMap that the HelmRelease references:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
resources:
  - ./helmrelease.yaml
configMapGenerator:
  - name: podinfo-helm-values
    files:
      - values.yaml=./helm-values.yaml
generatorOptions:
  disableNameSuffixHash: true

Finally, I add the new folder to the default namespace’s kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./podinfo

Since default already exists, there’s no namespace.yaml to create here. For namespaces like media or cnpg-system I’d add one alongside the apps. Once I commit and push these files, Flux notices the change, pulls the chart and deploys podinfo, no helm install in sight.

Setting up a cluster, start to finish

Spinning up the whole cluster from scratch comes down to running through a set of commands in order. Here’s the full process, combining the Talos steps from the previous post with the Flux setup from this one.

Make sure you’ve got the following tools installed first: talosctl, talhelper, kubectl, helm, sops and age.

Then, to set up the cluster itself:

  1. Update the talos/talconfig.yaml file to match your environment.
  2. From the talos/ directory, generate the Talos machine configurations: talhelper genconfig.
  3. For each machine, apply its configuration: talhelper gencommand apply --node <MACHINE_IP> --extra-flags="--insecure" | bash.
  4. Wait for the machines to reboot…
  5. Bootstrap the cluster: talhelper gencommand bootstrap | bash.
  6. Fetch the kubeconfig: talhelper gencommand kubeconfig --extra-flags="clusterconfig --force" | bash.
  7. Set the retrieved kubeconfig as the one kubectl should use: export KUBECONFIG=$(pwd)/talos/clusterconfig/kubeconfig.
  8. Install Cilium as the CNI:
kubectl label ns kube-system pod-security.kubernetes.io/enforce=privileged
helm repo add cilium https://helm.cilium.io/
helm install cilium --namespace kube-system cilium/cilium \
  --set ipam.mode=kubernetes \
  --set kubeProxyReplacement=true \
  --set operator.replicas=1 \
  --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
  --set cgroup.autoMount.enabled=false \
  --set cgroup.hostRoot=/sys/fs/cgroup \
  --set l2announcements.enabled=true \
  --set externalIPs.enabled=true \
  --set ingressController.enabled=true \
  --set ingressController.default=true \
  --set k8sServiceHost=localhost \
  --set k8sServicePort=7445

That last step is worth a quick note, Cilium gets installed by hand once so the cluster actually has networking before any pods (including Flux’s own) can schedule. It’s a bit of a chicken-and-egg situation. From then on, the HelmRelease in kubernetes/apps/kube-system/cilium adopts that release and keeps it in sync, so it’s managed by Flux like everything else. I go into this command in more detail in the previous post.

And then, to get Flux going:

  1. Generate an age key and add its public key to .sops.yaml (see the SOPS section above).
  2. Create the cluster-secrets.sops.yaml and github-deploy-key.sops.yaml files, then encrypt them with sops --encrypt --in-place.
  3. Adjust the repository link in kubernetes/flux/config/cluster.yaml to point at your own repository.
  4. Commit and push your changes, this is important, since Flux pulls from Git and can’t apply what isn’t pushed yet.
  5. Run through the bootstrap commands from the “Bootstrapping Flux” section above to install Flux and hand control over to it.

After that, Flux takes over and installs everything defined in the repository, taking the cluster from a set of bare machines to a fully running, GitOps-managed setup.

Next up

Now that Flux is running and workloads deploy themselves straight from Git, the cluster is in a state I’m actually happy to build on. Running all this by hand gets old fast though, so the next step is wrapping the whole thing in Taskfiles to get the setup down to a handful of task commands. After that I want to dig into the workloads themselves, Longhorn for storage, cert-manager for certificates and the apps I actually run day to day. We’ll see how short I can keep that post, this one definitely wasn’t short.