otche-backend/docs/CONFIG.md

Worker configuration and qualification proofs

Use provisioning/runtime-config.example.json as the complete base; replace metadata and paths, never copy an invented passed:true proof. Runtime reads this file only in worker/watchdog. All token/CA/proof paths are absolute inside their containers; seals belong in writable /var/lib/otche/source-seals/, not read-only /run/otche.

Credentials

Each owner's five distinct files contain { "token_id": "service@pve!worker", "secret": "secret-manager-value" }. Tokens are scoped separately; do not commit these files. Each owner has its own pool and private ISO storage. Distinct names backed by the same directory are not isolation. User/token effective permissions are intersected. provisioning/onboard-owner.py --help describes repeated --source-vmid and optional --extractor-source-vmid; dry-run is default. Use --apply only on reviewed private resources.

The supported single-account deployment uses existing otche@pve (firstname/comment otche, no password) with five separate privsep=1 tokens. Pass --service-user otche@pve to trusted onboarding. The account must already be enabled, non-expiring, and have no groups, tokens, ACL entries or effective resource privileges; an existing privilege-bearing account is refused, not reset. Token names include the owner prefix and purpose. The parent user receives only the union of the reviewed per-token grants on explicit paths; each token receives only its purpose's subset. No ACL is granted on /, /vms, /storage, or /access. Keep the account password unset: a password login would expose the parent union rather than a single token's subset.

The control container calls https://pve.example.invalid:8006/api2/json using Authorization: PVEAPIToken=<token_id>=<secret>, with the cluster CA and certificate IP/hostname verification. This is an outbound HTTPS call, not a PVE password login, guest SSH, or mounted host filesystem. Only the worker/watchdog credential mount receives these files. The API, frontend, browser and Windows guest receive neither token nor PVE configuration; do not mount the worker directory into those services. Store credential files mode 0600, parent directories 0700, readable only by the worker UID and the deployment administrator; mount them read-only.

Onboarding refuses pool/role/user collisions and overlapping directory storage before writing credentials. After a partial apply, inspect the exact scoped resources rather than blindly repeating or resetting the user. Five file paths do not by themselves prove separation: confirm all five token IDs differ, privsep=1, and inspect each token's effective privileges through the actual verified-TLS CT-to-PVE transport. Read-only authentication evidence is not source/Windows qualification and must not be converted into a passed isolation proof.

bindings-sync authenticates all five credentials through the application's Go PVE client (GET /access/permissions) before examining isolation evidence or setting configured=true. An invalid/revoked runtime, recorder, uploader or housekeeping token blocks the binding even when provisioner works and files exist. Successful authentication alone does not open admission: missing/expired isolation proof, unavailable private storage and source qualification remain separate fail-closed gates.

PVE token-management safety must be checked independently of privilege separation; do not grant User.Modify or broad parent-user privileges. Authentication alone is not isolation or source qualification.

Directory image volumes are supported as storage:VMID/vm-VMID-disk-N.qcow2 (also raw/vmdk); both directory and filename must match the allocated VMID. ZFS/LVM-style storage:vm-VMID-disk-N remains supported. Source/base volumes, mismatched ownership and traversal are rejected. Validate these restrictions on disposable resources before enabling execution.

Job ISO labels use OT plus the first 14 uppercase hexadecimal digits of the job UUID: exactly 16 ASCII characters, preserved identically in both ISO9660 primary and Joliet supplementary volume descriptors and Windows VolumeLabel. The label only locates candidate media; job.json must still match the complete job UUID, original SHA-256 and filename. Do not shorten those identity checks to the label prefix.

Python 3.13 enables an optional X509_STRICT profile that rejects the older PVE cluster CA because it lacks a CA keyUsage extension. The isolation probe's verified_context explicitly omits only that optional strict profile, matching the Go/curl validation contract: CERT_REQUIRED, trusted chain, certificate dates and hostname checks remain enabled. Verify wrong CA and wrong hostname rejection against your own deployment. No -k, CERT_NONE, check_hostname=false, changed cluster CA or swallowed TLS failure is used. Curl Authorization must be provided through stdin/private config, never a secret-bearing -H argument.

provisioning/probe-owner-isolation.py performs real read-only allow/deny checks, refuses 404/500 as proof of denial, checks exact effective privilege key allowlists (permission values describe propagation, not whether granted), and requires trusted nonalias storage inventory. Example:

python3 provisioning/probe-owner-isolation.py \
  --config /secure/config.json \
  --owner 11111111-1111-4111-8111-111111111111 \
  --foreign-owner 22222222-2222-4222-8222-222222222222 \
  --own-disposable 8101 --foreign-disposable 8102 --source-vmid 7001 \
  --storage-inventory /secure/storage-inventory.json \
  --evidence /secure/owner-one-isolation-evidence.json \
  --proof /secure/owner-one-isolation-proof.json

8101/8102 are examples only: use actual independently created disposable probes, never production guests. Administrator storage inventory is obtained through the PVE read API /storage and protected locally; symlink/dataset aliases require independent operator inspection. The tool does not create probes, execute payloads, modify firewall or mutate VMs. Destructive deny tests are never run on production. Copy completed proof into the corresponding configured isolation_proof_file without changing its bytes. One-day expiry requires revalidation, not editing the date.

Proofs are deployment-specific, expire after one day and require real revalidation. No acceptance logs or pre-passed proofs ship with this repository.

Binding metadata exposes isolation_expires_at (RFC3339 or null). API admission, dashboard and admin binding status evaluate expiry at request time; the worker also clears stale stored readiness. Null is unvalidated, and expired proof requires fresh real probes, never an edited expiry date. Source qualification is a separate required step on disposable clones. No published deployment state is implied by the example configuration.

Readiness now uses the signed fixed Otche-DesktopReady Limited/Interactive task, with a fresh nonce, bounded response and deadline, matching boot/account/console session, unlocked Default input desktop, real Explorer shell/taskbar and no visible setup/sign-in host. Resident UserOOBEBroker or generic WWAHost processes alone are not blockers. During a sample, the worker uses -BootOnly; post-run it uses -BaselineOnly, so sample-owned GUI is not reclassified as setup and no repeated interactive probe is launched. Capture and review your own protected source/fresh-clone evidence. Observed Defender and Windows versions are recorded, never inferred from marketing release names.

Parallel execution

Top-level concurrent is the global live-attempt cap (1–16, default 1). Each owner binding has max_active_attempts (1–16, default 1), which must not exceed concurrent. A binding omitted from owners cannot claim work. Set both limits to 9 for nine simultaneous runs of one Job; leaving the owner limit at 1 deliberately preserves serial execution. Claims fill available slots each tick and order equal-age runs by their immutable profile_name, then run ID. A recovered lease preserves the original claimed_at timestamp.

Reservation admission and allocation insertion run in one transaction under the owner's lock. The VMID lock ends as soon as the reservation commits; only the source-specific lock spans the full clone and both source digest checks. Lock waits respect caller cancellation and have a 20-minute ceiling. Different sources can clone concurrently without weakening protected-VMID, ownership, isolation or qualification checks.

• Disk: max_disk_bytes is the source validation ceiling, not a charge per clone. Each Windows allocation reserves the measured sum of source virtual disks (including EFI/TPM). max_owned_disk_bytes includes every non-deleted allocation and extractor, including retained evidence. Storage admission additionally reserves all not-yet-materialized clones (reserved, clone_intent, cloning) against available bytes above min_storage_free_bytes; it does not subtract full logical disk capacity again for already-materialized thin clones. Thin provisioning is not a guarantee against later disk growth: size the underlying storage for the workload and retain free-space monitoring.
• Memory: a clone reserves source memory MiB plus 1 GiB for QEMU overhead. Active owner reservations must fit node total - min_memory_free_bytes - host baseline; the baseline is node used memory minus measured mem of running VMs recorded in that owner's pool. Stopped/held evidence keeps its disk reservation but no running-memory reservation. Existing live free-memory checks remain mandatory.
• Artifacts: owner quota includes inputs, retained artifacts, job media and non-deleted allocation reservations. Each new allocation reserves max_video_bytes + 3 * max_artifact_bytes, plus one artifact bucket when Grub is requested. The private artifact filesystem must keep min_artifact_free_bytes after all live artifact reservations, not just one encoder. The configured video ceiling must realistically cover the longest observation plus collection tail; measure actual recordings before reducing it.

For a 32-thread / approximately 125 GiB node, nine 8 GiB / 4-vCPU clones reserve approximately 81 GiB of memory (9 × (8 + 1)), leaving room for a measured host baseline and, for example, 12 GiB minimum free memory. Their 36 virtual CPUs are overcommitted against 32 hardware threads: benchmark payload and encoder contention rather than treating the cap as guaranteed throughput. Set max_owned_disk_bytes >= 9 × measured source bytes + retained evidence headroom (and extractor capacity when enabled). Nine measured 64 GiB sources need at least 576 GiB before EFI/TPM, extractors and retained evidence; a 160 GiB owner quota cannot admit them. Available clone storage must separately satisfy the pending-clone check.

For an illustrative max_video_bytes = 512 MiB, max_artifact_bytes = 64 MiB and no Grub, nine runs reserve 6.19 GiB of artifacts before existing uploads/media/evidence and the filesystem free-space floor. This video limit is an example, not a measured bound: long or high-motion captures may require more. Give the worker container enough CPU for nine ffmpeg processes; a 2-CPU container cap can bottleneck recording even if the Windows guests have ample node CPU. Adjust the deployment's WORKER_CPUS, memory and PID limits after measuring, and restart the worker/watchdog when changing runtime configuration. The shipped example remains conservative at one active attempt.

Optional online policy

Offline remains the default (online: null in the example). Sources must be ordinary stopped VMs without any NIC. There is no inherited online bridge or firewall-hash policy. The privileged host broker attaches net0 only to an adopted, owned disposable clone; the worker never attaches networking to a source or receives host execution privileges.

To enable online qualification, explicitly configure the owner's private worker/watchdog binding:

{
  "online": {
    "endpoint": "https://network-broker.example.invalid:9443",
    "ca_file": "/run/otche/network-ca.pem",
    "certificate_file": "/run/otche/owner-one/network-client.pem",
    "key_file": "/run/otche/owner-one/network-client-key.pem",
    "proof_file": "/run/otche/owner-one/online-proof.json",
    "qualification_allowed_cidrs": ["1.1.1.1/32"]
  }
}

The qualification destination above is illustrative, not a sufficient Windows/Defender connectivity prescription. Choose and verify the actual operator-approved destinations. Qualification uses this explicit canonical list for both online EICAR and benign controls, with the same Defender baseline comparison as offline. It does not add exclusions or bypass background Windows traffic restrictions. Both actual online controls must pass before online_available is set.

The broker endpoint is an HTTPS origin without a path, query or embedded credentials. All four file paths are absolute; pin the broker server CA and issue a separate client certificate whose CN maps to this owner in the host configuration. Keep the private key and proof mount exclusive to worker/watchdog, never API, frontend or guest. Redirects, system proxy forwarding, untrusted servers and missing client authentication are rejected.

Operator-produced online proof fields are:

• owner_id, node, endpoint: exact owner/broker identities.
• passed, management_denied, default_denied, ipv6_denied, spoofing_denied, cross_clone_denied: all must be true results of real disposable-clone probes.
• expires_at: future RFC3339 expiry requiring real revalidation, not manual date extension.
• evidence_sha256: lowercase SHA-256 of protected actual probe evidence.

No pre-passed proof is provided. bindings-sync independently checks the proof and authenticated broker health, publishes online_ready, online_expires_at and a bounded online_reason, without changing offline readiness. API admission requires current online readiness/proof expiry in addition to profile qualification. Runtime rechecks proof and actual allocation before boot/dispatch and during execution; health alone is not isolation evidence. The obsolete fixed bridge/firewall hashes are not accepted as proof.

For operator-approved automation, provisioning/revalidate-owner.py, revalidate-network.py and probe-network-packets.py run new scoped authorization, Windows guest and isolated kernel probes. The deployment repository owns the idle gate, temporary clone lifecycle, schedule, evidence publication and recovery (revalidate.py and its systemd units). These are root-only PVE tools, not worker/API permissions. They consume private operator configuration and refuse protected sources as probe targets. Install the executable network helper from Git with its mode preserved because its fixed DHCP hook invokes the same file. No script turns historical evidence into a newly dated proof; failed checks or incomplete cleanup produce no passed proof. Windows source qualification remains separate.

Online jobs require settings.allowed_cidrs: 1–128 IPv4 addresses/CIDRs, at most 8192 input UTF-8 bytes. Bare addresses become /32, hostbits are masked, duplicates removed preserving order. DNS names, ranges, IPv6, blank entries, unusable special prefixes and mixed private/public broad prefixes are rejected. 0.0.0.0 and 0.0.0.0/0 mean public IPv4 only, never private/local/special networks. Explicit private/lab CIDRs are separate opt-ins and remain alongside the public wildcard. Protected infrastructure and the broker's sandbox pool always remain denied, even explicitly listed. Offline jobs have no nonempty list; historical online jobs without a list are not silently granted access.

The worker persists the broker request before /v1/prepare, using the allocation UUID as durable lease_id, then saves the complete validated response under allocation.metadata.network. An immutable network-policy.json artifact captures the canonical list, digest, address/MAC/bridge assignment and initial broker expiry. It does not contain private keys. Every /v1/check must match owner, attempt, VMID, node, lease, policy digest and original topology. The actual clone must have exactly the broker-issued VirtIO MAC, bridge and firewall=1, with no extra NIC or VLAN option.

Renewal runs independently of QGA every 10 seconds starting immediately after prepare, through boot, readiness, delivery, observation and collection. A failed renewal cancels execution with an explicit network-policy error, tries immediate revocation and then follows stop/release cleanup. The broker's kernel active-port timeout is 45 seconds: worker/broker failure does not leave established connections authorized. No automatic re-prepare, lease replacement or command redispatch occurs. Stop, watchdog, retained-evidence release and deletion use the recorded identity; they release only after confirmed stopped/absent state, before deletion, and refuse changed/reused identities. Ambiguous prepare responses still have a durable release request.

DNS has no implicit bypass: DHCP advertises 1.1.1.1 only when that address is effectively allowed; otherwise it advertises no resolver. There is no host/system resolver fallback, DNS relay or IPv6 fallback. An allowed destination is not a promise that its service will answer. All user and Windows/Defender background traffic is constrained to the same list, including when the payload runs as guest administrator.

Optional read-only extractor

Top-level configuration:

{
  "extractor": {
    "node": "pve-node",
    "source_vmid": 7101,
    "config_digest": "ACTUAL_PVE_CONFIG_DIGEST",
    "proof_file": "/run/otche/extractor-proof.json",
    "source_disk": "scsi0",
    "max_disk_bytes": 17179869184,
    "include_crash_dump": true,
    "timeout_seconds": 180
  }
}

7101 is an example, not a discovered VM. source_disk is the owned Windows disk to extract (default scsi0, preserving system evidence); max_disk_bytes is reserved space for the full Linux extractor clone. timeout_seconds must 30–300. Source is stopped ordinary Linux, no NIC/USB/PCI passthrough, scsi1 empty, QGA installed, fixed otche-extract installed through install-extractor.sh. No disk is formatted by these tools.

Extractor proof fields: passed, node, source_vmid, config_digest, expires_at, evidence_sha256, read_only_verified, no_egress. Require actual read-only reassignment verification (PVE ro flag, Linux blockdev RO, attempted writes denied on benign disposable evidence), exact serial+size, no egress, bounded QGA export. The runtime does not invent or rewrite this proof.

The worker journals source/destination ownership, task IDs and disk moves in extractor_allocations. Before boot, borrowed evidence is ro=1 and gets an allocation-bound serial. No pending config is accepted. Original Windows VM and its disks remain retained on collection failure, including missing config, encryption, dirty filesystem, timeout and ambiguous UPID. The Linux VM uses ntfs-3g ro,norecover; neither PVE nor control CT mounts NTFS. It stops externally on completion/failure. Evidence must return read-only to the original stopped Windows VM before extractor deletion; unresolved states remain journaled for watchdog/operator.

Enable execution only after configuration

docker compose --profile execution run --rm worker bindings-sync
docker compose --profile execution run --rm worker source-maintenance --source-ref win11-stopped-revision
# Operator performs approved source maintenance and graceful shutdown separately.
docker compose --profile execution run --rm worker source-validate \
  --profile-id ACTUAL_PROFILE_UUID --source-ref win11-stopped-revision
docker compose --profile execution up -d worker watchdog

After source validation, admin queues qualification, reviews actual evidence, and publishes only a passed immutable revision. Missing account credentials/script policy/signing trust, stale Defender or inaccessible QGA remain actionable prerequisites. Never bypass protections or change a queued source revision to make admission pass.

Windows JSON observation and partial detections

Mutable Windows JSON is read through one bounded, read-only PowerShell snapshot launched over QGA. The filename is encoded data, not shell syntax; the reader opens with FileShare.ReadWrite | FileShare.Delete, reads at most 4 MiB from one file generation and closes before returning base64 output. It does not execute file contents, bypass signing policy, or retry an ambiguous launch. Binary/log transport keeps its existing bounded QGA file reads. Holding guest-file-open across transport round trips can otherwise block the signed runner's atomic File.Replace even after its finite contention retries.

On abnormal completion the worker retains the incomplete outcome and telemetry status, but separately salvages positively correlated Defender detections from the collector. Receipt command/attempt/job, boot, timestamps and exact sample resource must match; old/unrelated detections never imply a verdict, and missing evidence never implies clean. Already captured final reports and positive detections are not overwritten by stale runner status. Historical attempts are not rewritten by this change.

Grub artifact collection

settings.grub_paths is optional immutable Job data, not a worker host path. The interactive runner captures literal local files under the already-selected user/admin token after observation; the SYSTEM telemetry collector exports only fixed numbered staged files into the protected control directory after no-reparse/hardlink and hash/size validation. It never opens requested Grub paths as SYSTEM. Requested paths are never resolved on the CT, PVE host or workstation, nor interpolated into shell source.

Grub reserves one additional max_artifact_bytes bucket in the existing owner artifact quota and headroom check. Payload limits are min(8 MiB,max_artifact_bytes/2) per file and min(32 MiB,max_artifact_bytes/2) total. Snapshot and protected staging passes each use at most half collect_timeout_seconds; the worker still applies its existing collection/safety deadlines. Staged files are not individually published: the controller streams one ZIP, hashes transported bytes, writes a controller-owned manifest and atomically publishes ZIP metadata plus validated report under the current attempt lease. A unique exclusive private archive path prevents replacing earlier immutable evidence. Ambiguous commit preserves bytes.

Ordinary per-file errors and best-effort changed snapshots make Grub partial/empty, independently of Defender verdict and telemetry. They do not by themselves retain clones. Missing export, corruption, timeout or private persistence failure does retain stopped evidence. No Grub archive is created for historical jobs without paths. Guest administrator tampering remains within the documented untrusted guest-evidence boundary; Grub files must never be treated as trusted code.

Manual Windows names and technical telemetry

Profile names are fixed, manually editable full labels. Include the desired build literally when naming an environment; there is no automatic build suffix or build override layer. New normal and qualification Runs snapshot that exact name, and later profile edits do not rename historical Runs. Legacy migration freezes the currently known name without inferring past builds. The signed baseline still captures actual CurrentBuildNumber, UBR and BuildLabEx in environment.os_build for fingerprint/drift checks and technical reports only; telemetry never rewrites display names.