Technology

K3s Known Issues and Hardened Cluster Upgrade Guide

byAmelia Emma

July 27, 2023

1.4K views

2 minute read

Image by: https://docs.k3s.io/

Introduction: ⁠

Rancher Labs’ K3s, a simplified Kubernetes distribution, is being ⁠ embraced for its simplicity and resource efficiency. K3s users may face known issues when deploying it ⁠ in production settings, as with any program. The following section outlines the upgrade method for upgrading clusters from v1.24.x to ⁠ v1.25.x in K3s, which also emphasises the major issues affecting the project. ‍

Known Issues:

Snap Docker: ‍

Using K3s with Docker necessitates the avoidance of ⁠ installing Docker using a snap package. When using Docker from snap, it is known ⁠ that difficulties can occur when running K3s. Flawless execution requires Docker downloaded ⁠ from other reliable sources. ‌

Iptables and nftables Mode:

Using K3s may lead to complications while employing iptables ⁠ in nftables mode rather than legacy mode. Using the newest iptables is recommended because it ⁠ is less likely to disrupt K3s.

Rootless Mode: ‌

Running K3s in Rootless mode should be attempted with care ⁠ because it could potentially have several bugs and problems. Users must validate that this mode is dependable and ⁠ trustworthy when used in a production environment. ‍

known issues — Image by: https://akyriako.medium.com/

Upgrading Hardened Clusters from ⁠ v1.24.x to v1.25.x:

Kubernetes has omitted the PodSecurityPolicy (PSP) feature from ⁠ v1.25 to utilize Pod Security Standards (PSS). K3s users with PSP configurations must perform specific ⁠ manual steps during the upgrade process. Here’s a step-by-step guide: ‌

Update kube-apiserver-arg Value:

The kube-apiserver-arg value on every node needs to ⁠ be updated to eliminate the PodSecurityPolicy admission-plugin. Instead, add the following ⁠ arg value: ‍

arduino

'admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml'

Do not turn on or ⁠ update K3s instantly.

Create psa.yaml File: ‍

Prepare the /var/lib/rancher/k3s/server/psa.yaml file with ⁠ the text presented below. Customize exemptions as per ⁠ your requirements:

yaml

apiVersion: apiserver.config.k8s.io/v1

kind: AdmissionConfiguration

plugins:

- name: PodSecurity

  configuration:

    apiVersion: pod-security.admission.config.k8s.io/v1beta1

    kind: PodSecurityConfiguration

    defaults:

      enforce: "restricted"

      enforce-version: "latest"

      audit: "restricted"

      audit-version: "latest"

      warn: "restricted"

      warn-version: "latest"

    exemptions:

      usernames: []
      runtimeClasses: []
      namespaces: [kube-system, cis-operator-system, system-upgrade]

These names represent the cluster system, ⁠ Cisco Operator system, and system-upgrade

Perform the Upgrade:

Once the kube-apiserver-arg is updated and the psa.yaml file is generated, ⁠ the K3s upgrade should be done in the ordinary fashion.

Remove Remaining PSP Resources: 

Upon completion of the upgrade process, eliminate all PodSecurityPolicies ⁠ and the connected RBAC resources from the cluster. You can use the following commands to identify ⁠ and delete the resources associated with PSPs: ‌

shell

# Get the resources associated with PSPs

$ kubectl get roles,clusterroles,rolebindings,clusterrolebindings -A | grep -i psp



# Delete those resources:

$ kubectl delete clusterrole.rbac.authorization.k8s.io/psp:restricted-psp cluste

Conclusion: ‍

A smooth K3s deployment is achieved by learning about known ⁠ issues and taking the necessary steps to harden clusters. To guarantee that their K3s clusters remain stable, safe, and up to date with ⁠ the most recent Pod Security Standards, users must adhere to these guidelines.