Development

Note

Before anything else, please read the contributing guide: it covers the project status, the contribution process and the requirements for pull requests to be accepted.

kuik uses cert-manager to issue and configure its webhook certificate, so you need it installed on your cluster first.

Commands are Task tasks; task --list shows them all.

Deploy on a Kind cluster

The Helm chart is the only deployment path, for development as for production.

kind create cluster --name kuik-dev
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
# the chart creates a Certificate and an Issuer, which cert-manager's webhook must be ready to admit
kubectl wait --for=condition=Available deployment --all -n cert-manager --timeout=300s
# build the image, load it into the Kind cluster and install the chart in kuik-system
task kind-deploy IMG=kuik:dev KIND_CLUSTER=kuik-dev

Tip

Kind nodes are containers sharing the host kernel, and a cluster consumes many inotify instances. On the distribution defaults, pods fail with failed to create fsnotify watcher: too many open files, especially when several clusters run at once. Raise the limits on the host (see the Kind known issues):

sudo sysctl fs.inotify.max_user_instances=512 fs.inotify.max_user_watches=524288

Persist them in /etc/sysctl.d/ to survive a reboot.

task deploy IMG=... alone runs helm upgrade --install against the current kubeconfig, with, when HELM_VALUES names a file, the values it holds; arguments after -- go to Helm (task deploy -- --set verbosity=DEBUG). task undeploy removes the release, task uninstall the CRDs.

Run the manager on your host

# generate CRDs definitions from go code and install them on the cluster you're connected to
task install
# run one process locally against the cluster you're connected to and export metrics to :8080
task run:reconciler

kuik runs as three separate processes, one subcommand each: webhook, reconciler and secret-syncer. They are split by privilege and availability, they exchange nothing directly, and the chart deploys one Deployment per process. Run one with task run:<process>, and the three side by side with task run, which starts them in the same terminal and stops them together.

The webhook is the one that needs setting up: it serves TLS, so it wants a certificate in /tmp/k8s-webhook-server/serving-certs and a way for the API server to reach your host, see Local webhook for remote cluster. The other two need nothing beyond your kubeconfig, which is why the reconciler is the usual starting point. The reconciler creates the ConfigMap kuik-cluster-id in the cluster resource namespace (kuik-system by default), so create that namespace first (kubectl create namespace kuik-system); the webhook stays unready until the reconciler has run once.

Each process binds its own metrics and probe ports, so they can run side by side: metrics on METRICS_PORT (8080 by default) and probes on HEALTH_PORT (8090), plus 1 and plus 2 for the second and third process started by task run.

task run options

task run can be configured through variables, given as environment variables or as task run VAR=value:

• RUN_FLAG_DEVEL: sets the -zap-devel flag, defaults to true
• RUN_FLAG_CONFIG: the global config file the processes load and reload, defaults to hack/dev-config.yaml
• RUN_FLAG_LOG_LEVEL: sets the -zap-log-level flag if present
• RUN_FLAG_ZAP_ENCODER: sets the -zap-encoder flag if present
• METRICS_PORT: first port to bind for the metrics, defaults to 8080
• HEALTH_PORT: first port to bind for the health probes, defaults to 8090
• RUN_ADDITIONAL_ARGS: add any additional argument to the go run ./cmd command (you can even | grep here)

They apply to task run and to the three task run:<process> tasks alike. Arguments after -- are passed to the manager as well: task run:reconciler -- -zap-log-level=debug.

I highly suggest that you try github.com/pamburus/hl, an awesome tool to make json logs human readable. It can be setup with kuik like this:

export RUN_FLAG_ZAP_ENCODER=json RUN_ADDITIONAL_ARGS="2>&1 | hl --paging=never"
task run

To read the logs of a deployed kuik, from the command line, Sofka or k9s, see Reading the logs.

Grant a process a permission

Each process runs under its own ServiceAccount and holds only its own rules. Declare a permission with a // +kubebuilder:rbac marker on the code that uses it, naming its process with roleName=webhook, roleName=reconciler or roleName=secret-syncer (namespace=kuik-system for a Role in the install namespace), then run task manifests. The chart binds the generated roles to the ServiceAccounts, and helm template fails on a marker without roleName=.

Local webhook for remote cluster

There are several ways of developing a webhook for kubernetes and depending on your situation you may prefer one over another. One of them consists of running your webhook locally (using task run:webhook) and expose it as a service in your kubernetes cluster using a tool like github.com/omrikiei/ktunnel for instance. Since MutatingWebhookConfiguration requires a certificate for authentication, you will need to create one using cert-manager.

You will need:

• To install ktunnel
• To install cert-manager
• To create a tunnel with ktunnel (see script below)
• To issue a Certificate for your mutating webhook
• To copy this certificate locally for you dev instance of kuik to use it
• To create a MutatingWebhookConfiguration using the service that ktunnel create for you

Here is a helper script that reads your mutating webhook certificate and place it somewhere kuik will find. It also create a service and setup the tunnel using ktunnel:

#!/bin/bash

NAMESPACE=kuik-system
SECRET=webhook-server-cert
SERVICE=webhook-service
PORTMAP=9443:9443

kubectl -n "$NAMESPACE" get secret "$SECRET" -o jsonpath="{.data['tls\.key']}" | base64 --decode > tls.key
kubectl -n "$NAMESPACE" get secret "$SECRET" -o jsonpath="{.data['tls\.crt']}" | base64 --decode > tls.crt

mkdir -p /tmp/k8s-webhook-server/serving-certs
mv tls.* /tmp/k8s-webhook-server/serving-certs/

kubectl tunnel expose -n "$NAMESPACE" "$SERVICE" "$PORTMAP" -r

And here is the MutatingWebhookConfiguration with required CertificateRequest et CertificateIssuer:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: kuik-system/webhook-server-cert
  name: mutating-webhook-configuration
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: webhook-service
      namespace: kuik-system
      path: /mutate--v1-pod
      port: 9443
  failurePolicy: Ignore
  reinvocationPolicy: IfNeeded
  name: mpod-v1.kb.io
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
  sideEffects: NoneOnDryRun

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: webhook-server-cert
  namespace: kuik-system
spec:
  dnsNames:
  - webhook-service.kuik-system.svc
  - webhook-service.kuik-system.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: selfsigned-issuer
  secretName: webhook-server-cert

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: kuik-system
spec:
  selfSigned: {}

Simulate unreachable registry with a CiliumNetworkPolicy

If you have kube-image-keeper running in a cluster with Cilium as CNI, you can easily simulate an unreachable registry with a CiliumNetworkPolicy targeting the kuik manager (which perform the active check to know if an image is available).

For example:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: test-kuik-v2
  namespace: kuik-system
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/instance: kube-image-keeper
  ingress:
    - fromEntities:
      - kube-apiserver
    - fromEndpoints:
        - {}
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: monitoring
      toPorts:
        - ports:
            - port: "8080"
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*"
    - toEntities:
      - kube-apiserver
    - toEndpoints:
        - {}
    - toFQDNs:
      # DockerHub
      - matchName: docker.io
      - matchPattern: '*.docker.io'
      # Quay
      - matchName: quay.io
      - matchPattern: '*.quay.io'
      # AWS Public ECR
      - matchName: public.ecr.aws
      - matchPattern: '*.cloudfront.net'
      # Github Container Registry
      - matchName: ghcr.io
      # Custom registry
      - matchName: my-registry.company.com

⚠️ When you configure a Network Policy, ingress and egress traffic for the endpointSelector will be denied by default and only those defined will be allowed. In this example we allow:

• ingress from apiserver
• ingress from namespace monitoring to metrics port
• egress to DNS service
• egress to apiserver
• egress from some FQDN to allow specific registries

Any registry (including its redirections) not present in the toFQDNs list will not be reachable by kuik and will result in a registry unreachable. So you can play with this to quickly simulate unreachable registry without impacting your whole cluster.