Cluster security

The Cluster tab of a workspace follows your Kubernetes clusters. On every node, mlab-agent runs k3sec as its tool: k3sec watches the node (Kubernetes events, process executions, outbound connections, YARA detections) and lists the packages of every running image, tagged by what is really loaded in memory. mlab.sh matches them against known vulnerabilities and shows you, per cluster, what to fix first.

k3sec holds no mlab.sh key and opens no connection of its own: the agent of the node owns the identity and the only outbound link, as on any other host.

Set it up

One pod per node, from a DaemonSet: the image ghcr.io/mlab-sh/k3sec holds k3sec, mlab-agent and the default YARA rules.

kubectl create namespace k3sec
kubectl -n k3sec create secret generic k3sec-enroll --from-literal=code=<code>

curl -fsSLO https://raw.githubusercontent.com/mlab-sh/k3sec/main/deploy/kubernetes/k3sec.yaml

kubectl apply -f k3sec.yaml
kubectl -n k3sec get pods -o wide

Updating

Point the DaemonSet at the new image:

kubectl -n k3sec set image ds/k3sec k3sec=ghcr.io/mlab-sh/k3sec:<version>
kubectl -n k3sec rollout status ds/k3sec

Unlike an agent installed on a host, there is no policy to edit: each pod pins the SHA-256 of the k3sec binary of its own image when it starts. Nodes keep their keys and do not enrol again. Releases are listed on GitHub, with their changes.

What the pods run as

tracefs must be mounted on the node at /sys/kernel/tracing (systemd does it on most distributions). Without it, the node reports its eBPF source as failed and keeps the others running.

The node policy

The k3sec-policy ConfigMap is the agent's local policy on every node: what mlab.sh may ever ask. The workspace chooses within it, never beyond.

enabled = true                                   # kill switch
upload_hosts = ["<upload host from the Policy editor>"]

[tools.k3sec]
path = "/usr/bin/k3sec"
sha256 = "@K3SEC_SHA256@"                        # filled at start with the image's binary
enabled = true
allowed_ops = ["status", "configure", "events_since", "inventory", "sbom"]

[tools.k3sec.settings]
node = "@NODE_NAME@"                             # filled at start
allow_k8s_events = true
allow_logs = false                               # pod logs stay in the cluster, whatever mlab.sh asks
allow_ebpf = true
allow_rules = true
allow_vulns = true
namespaces = []                                  # empty: all; otherwise mlab.sh can only narrow this list
rulesets = ["/usr/share/k3sec/rules"]
allow_remote_rules = false                       # YARA rules sent from mlab.sh

All settings: k3sec › Settings. After editing the ConfigMap, restart the pods (kubectl -n k3sec rollout restart ds/k3sec).

The Cluster tab

The workspace's Cluster tab lists your clusters, each with its nodes online, images, vulnerabilities in loaded code and known-exploited ones. Click one for its dashboard.

Reachability

Every package of a running image is tagged:

A dormant package that starts running (a shell opened in a container, a debug tool) becomes loaded at its first execution, and its vulnerabilities are reported then.

What the cluster reports

The Config tab chooses, for the whole cluster: the sources (Kubernetes events, pod logs, eBPF events, YARA scanning, inventory and vulnerabilities), the namespaces, the lowest severity reported, and the YARA rule set. Push to the cluster sends it to every node, which applies it at once without restarting. Each node keeps its policy: what it does not allow stays off, and the table below the form shows what each node really runs (for example "pod logs not allowed by the nodes' policy"). Only owners and admins can push a configuration.

Data

Events are numbered by each node. If some are missed (a lost connection), mlab.sh asks the node for them again; what the node no longer holds is counted as lost, never silently dropped.

Limits

• YARA on executions scans the executable. A script is scanned as its interpreter (/bin/sh), not as its text. A binary that runs for a millisecond is still scanned, through another process of its container.
• Nodes running inside containers (k3d, kind, k3s in Docker): the PIDs the kernel reports do not exist in the node's view. Events still reach their pod, and YARA on executions and the dormant to loaded promotion go through another process of the container; YARA on written files sees nothing there. Real nodes and virtual machines are not affected.
• Reachability is a snapshot, taken at each inventory and updated by executions: a library loaded later by a running process shows dormant until the next inventory.
• Binaries no package owns (servers built from source into /usr/local) are not matched themselves, only the libraries they load. Go binaries are, through their build information.

Troubleshooting