assay

Assay the weights before you trust them.

assay is an offline-first, single-binary scanner for ML model artifacts (safetensors, GGUF, PyTorch pickle). It answers two questions about a model you just downloaded:

1. Is this file what it claims to be? - provenance & integrity
2. Does loading it put my machine at risk? - format-level safety

A downloaded model is a multi-gigabyte opaque blob that people execute with total trust. We would never do that with a random .exe. assay applies the same supply-chain hygiene to model weights.

Two-phase architecture

Phase 1 - provenance & integrity

Default, always on. Boring, solid, high-confidence, deterministic verdicts:

• Format detection & structural parsing (safetensors / GGUF / pickle)
• Pickle / arbitrary-code-execution risk flagging
• safetensors header & offset validation
• GGUF metadata sanity + embedded-template flagging
• Deterministic content hashing (per-tensor + manifest)
• Signature verification (detached ed25519 + model-transparency manifest; full Sigstore/cosign chain reported as unverified, not trusted)
• Human + JSON reports, CI-friendly exit codes

Phase 2 - weight inspection

Opt-in via --deep. Inspects the weights themselves. There is no external ground truth here, so Phase 2 emits signals with scores and severities, never verdicts - a high score means "anomalous, worth a human look", never "malicious". It never loads or executes the model; tensors are read cold and streamed (mmap, online moments) so peak RAM stays well under model size.

compare SUBJECT BASELINE - differential weight analysis

Weight analysis is most honest as a diff against a known-good reference, not a judgment of a model in isolation: a normally-trained transformer is naturally non-uniform across layers, so the standalone profile can flag legitimate peaks. compare makes the baseline the zero line - identical models are silent, a uniform fine-tune shows broad even drift (quiet), and a localized tamper shows a single concentrated spike (flagged). It guards against cross-architecture comparison (ARCH_MISMATCH, override with --force) and emits STRUCTURAL_DIVERGENCE for added/removed/reshaped tensors, LAYER_DRIFT_OUTLIER / TENSOR_DRIFT for concentrated drift, and IDENTICAL when drift is ~0 everywhere.

Installation

# from crates.io
cargo install assay

# or grab prebuilt static binaries from GitHub releases (no runtime deps, single file)

Usage

# scan a single file
assay scan model.safetensors

# scan a whole model directory (HF-style repo)
assay scan ./Qwen2.5-0.5B-Instruct/

# CI mode: machine-readable, non-zero exit on findings
assay scan ./model/ --json --fail-on high

# verify a signature / provenance bundle alongside the weights
assay verify ./model/ --bundle model.sig

# Phase 2: inspect the weights (signals, not verdicts)
assay scan ./model/ --deep --profile          # per-tensor stats + layer sparkline
assay scan ./model/ --deep --svg profile.svg  # write the 1D layer-profile chart
assay scan ./model/ --deep --mad-k 5.0 --json # tune the robust anomaly threshold

# compare: how a model differs from a known-good baseline (the honest profile)
assay compare ./model-suspect/ ./model-known-good/      # drift profile + spikes
assay compare ./subject/ ./baseline/ --svg drift.svg --json
assay compare ./a/ ./b/ --force                         # across architectures (unreliable)

Flags

Exit codes

Example output

scan --deep --profile on a real gpt2 repo (Phase 1 + Phase 2):

$ assay scan ./models/gpt2 --deep --profile
[1/2] ./models/gpt2/model.safetensors CLEAN 3 finding(s) (22.90s)
[2/2] ./models/gpt2/pytorch_model.bin UNTRUSTED 3 finding(s) (1ms)
✓ scanned 2 artifact(s) - 1 clean, 1 untrusted, 1.0 GiB in 22.91s

./models/gpt2/model.safetensors  [safetensors]  -> CLEAN
  manifest: blake3:d4ceed607f7040ba84b91eadef010d98079f9d9d85ffd6faf13d77ce958eccdf
  signature: unsigned
  [low]  WEIGHT_OUTLIER_LAYER: layer 3 is anomalous on mean_kurtosis (6.0 MADs from the cross-layer median)
  [info] ARCH_DETECTED: structural fingerprint: gpt2 (gpt2)

./models/gpt2/pytorch_model.bin  [pickle]  -> UNTRUSTED
  [high]   PICKLE_RCE_RISK: pickle artifact can execute code at load time
  [info]   SAFE_ALTERNATIVE_AVAILABLE: a safetensors artifact is present in the same repo; prefer it

The standalone profile flags layers 3 and 11 - but on a model in isolation you can't tell a legitimate peak from an injected one. That's exactly why compare exists: against a tampered copy, only the tampered layer spikes, and the innocent peaks stay silent.

What Phase 1 checks

• Format detection - identifies each artifact and refuses to guess. A repo mixing safetensors and pickle is itself a signal.
• Pickle / RCE risk (highest priority) - flags every pickle artifact as untrusted-by-default, runs an opcode-level scan for dangerous patterns (GLOBAL, REDUCE, imports of os / subprocess / builtins), and tells you whether a clean safetensors equivalent exists in the same repo.
• safetensors structural validation - parses the header and length prefix, validates every data_offsets [begin, end] (in-bounds, non-overlapping, no gaps), and rejects dtype/shape mismatches.
• GGUF metadata sanity - validates magic + version, tensor count and KV block, checks offsets, and flags embedded Jinja2 chat templates for human review.
• Deterministic hashing - per-tensor digest + manifest hash stable across re-containerization (renaming/repacking doesn't change identity).
• Signature / provenance verification - verifies a Sigstore bundle / cosign signature / model-transparency manifest against the computed hashes. Reports: signed / unsigned / signature-mismatch.

Design principles

• Offline-first. No network calls during a scan. Signature roots are bundled or supplied explicitly.
• Single static binary, no runtime deps. Drop it into a CI image or an air-gapped box and run.
• Honest confidence. Every finding carries a severity. Phase 1 is high-confidence by design; it never pretends to detect backdoors it can't.