k3sec
Runtime security for Kubernetes nodes: cluster events, eBPF, YARA and the vulnerabilities of the code that really runs. Under mlab-agent in a workspace, or standalone from the command line.
k3sec watches a Kubernetes node: cluster events and pod logs, process executions, outbound connections and sensitive file writes (eBPF), YARA detections on what runs, and the packages of every running container, tagged loaded or dormant depending on whether their code is in memory. Vulnerabilities are matched through vuln.mlab.sh, loaded code first.
In a workspace it runs under mlab-agent on every node of a cluster, and your team follows it from the Cluster tab. It also works standalone, from the command line. Source: github.com/mlab-sh/k3sec (Apache-2.0).
Linux x86_64, glibc 2.35 or later (Debian 12, Ubuntu 22.04, RHEL 9 and newer). The eBPF probe is built in and needs Linux 5.8 or later. Container images: Debian, Ubuntu, Alpine, Rocky and Alma are matched; others are inventoried with a warning.
Installation
In a cluster with a workspace, use the DaemonSet: Cluster security › Set it up. On a node, from the packages (see mlab-agent for the repository setup):
sudo apt install k3sec # Debian, Ubuntu (apt.mlab.sh)
sudo dnf install k3sec # Fedora, RHEL, Rocky, openSUSE (pkg.mlab.sh)The packages install /usr/bin/k3sec, the default YARA rules in /usr/share/k3sec/rules, and a k3sec.service unit (disabled) that runs watch --ebpf --rules on the host. Binaries and the signed SHA256SUMS are on the releases page; the container image is ghcr.io/mlab-sh/k3sec.
Commands
| Command | What it does |
|---|---|
k3sec watch | Streams cluster events in real time. --logs adds pod logs, --ebpf the kernel events (--kind exec,connect,file_write), --rules <dir> YARA detections on executed binaries and written files, --vulns the vulnerabilities of new pods. --rollback replays what is already there first. |
k3sec inventory | The packages of every container running on this node, with their reachability. --sbom <dir> writes one CycloneDX SBOM per image. |
k3sec vulns | Inventory plus vulnerability matching: CVE, severity, CVSS, known-exploited (CISA, EU), fixed version, reachability, grouped by image. --only-loaded, --kev-only, --min-severity <level>. |
k3sec serve | Started by mlab-agent, not by hand. |
Every command takes -n <namespace>, --node <name> (default $NODE_NAME: only this node's pods), --kubeconfig <path> and -o json. inventory, vulns and --ebpf need root, or the capabilities of the DaemonSet. Diagnostics go to stderr, output to stdout.
$ sudo k3sec vulns
11507 vulns, 335 loaded (1 KEV loaded) across 7 images
docker.io/library/nginx:1.25.3 (sha256:c7a6ad68be85) Debian GNU/Linux 12 (bookworm)
829 vulns, 104 loaded
LOADED KEV HIGH 7.5 CVE-2023-44487 nginx 1.25.3-1~bookworm
LOADED CRIT 9.8 CVE-2023-45853 zlib1g 1:1.2.13.dfsg-1Standalone, matching calls https://vuln.mlab.sh (--mlab-url for another one); set MLAB_API_KEY for larger nodes. In a workspace, matching happens in mlab.sh and k3sec reaches nothing but the Kubernetes API.
What it sees
| Source | Events |
|---|---|
| Cluster events | Every Kubernetes event of the watched namespaces, live. |
| Pod logs | Log lines of the pods, live. Off by default in a workspace. |
| eBPF | exec (every successful execution in a container), connect (outbound TCP, IPv4 and IPv6; not to 0.0.0.0 or loopback, which never leave the pod), file_write (every write-open, whatever the call: open, openat, creat, under /etc/, /usr/bin/, /root/, /var/run/secrets). Each event carries its pod, container, process and user. |
| YARA | The binary of each execution and each written file, scanned with your rules; a match is a critical detection. A binary that runs for a millisecond is read through another process of its container. A file found clean is not read again until it changes; a file that matched is scanned at every execution. |
| Vulnerabilities | The packages of each running image (dpkg, apk, rpm, language lockfiles, Go build information) and the vulnerabilities that apply, loaded code first. A dormant package that starts running is reported then. |
Settings (agent policy)
[tools.k3sec.settings] in the agent's policy (the k3sec-policy ConfigMap of the DaemonSet): the ceiling of what the workspace may turn on.
| Key | Default | Meaning |
|---|---|---|
allow_k8s_events | true | Cluster events may be reported |
allow_logs | false | Pod logs may be reported |
allow_ebpf | true | eBPF events may be reported |
allow_rules | true | YARA scanning may run |
allow_vulns | true | Inventories may be sent for vulnerability matching |
namespaces | [] | Namespaces that may be watched; empty means all |
rulesets | [] | YARA rule files or folders the workspace may pick from (by name: the last path component, .yar dropped) |
allow_remote_rules | false | The workspace may send its own YARA rules (compiled before use, no includes, no execution) |
node | unset | This node's name, so only its pods are watched (the DaemonSet fills it); unset, every pod of the cluster |
kubeconfig | in-cluster | A kubeconfig file instead of the pod's service account |
backlog_events | 50000 | Events kept on the node to resend after a lost connection (and at most 32 MiB) |
Allowed operations: status, configure, events_since, inventory and sbom (the last two upload, so the policy needs the upload host). Unknown settings are refused.
engram
Memory forensics for Windows dumps: the engram tool runs under mlab-agent in a workspace, or offline from the command line.
postmortem
Offline static dependency scanner for Node.js, Python, Rust, Ruby, PHP, Go and JVM projects. Resolves the lockfile graph, walks vendored sources, and flags supply-chain compromise patterns.