Publish reviewed standalone otche-backend sources
This commit is contained in:
omar
@@ -0,0 +1,26 @@
|
||||
**
|
||||
!go.mod
|
||||
!go.sum
|
||||
!cmd/
|
||||
!cmd/**
|
||||
!internal/
|
||||
!internal/**
|
||||
!windows/
|
||||
!windows/*.ps1
|
||||
!windows/*.psm1
|
||||
!windows/Source-Setup.txt
|
||||
**/secrets/**
|
||||
**/.env*
|
||||
**/.git-credentials
|
||||
**/.netrc
|
||||
**/.npmrc
|
||||
**/*.key
|
||||
**/*.pem
|
||||
**/*.pfx
|
||||
**/*.p12
|
||||
**/*.crt
|
||||
**/*.cer
|
||||
**/artifacts/**
|
||||
**/*.exe
|
||||
**/*.zip
|
||||
**/*.log
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
secrets/
|
||||
.runtime/
|
||||
artifacts/
|
||||
node_modules/
|
||||
dist/
|
||||
build/
|
||||
coverage/
|
||||
*.tsbuildinfo
|
||||
*.exe
|
||||
*.dll
|
||||
*.pfx
|
||||
*.p12
|
||||
*.key
|
||||
*.pem
|
||||
*.log
|
||||
*.zip
|
||||
*.iso
|
||||
*.mp4
|
||||
*.webm
|
||||
*.dump
|
||||
*.db
|
||||
*.sqlite*
|
||||
*.sql.gz
|
||||
__pycache__/
|
||||
.DS_Store
|
||||
|
||||
.git-credentials
|
||||
.netrc
|
||||
.npmrc
|
||||
*.crt
|
||||
*.cer
|
||||
*.p12
|
||||
coverage/
|
||||
.vite/
|
||||
playwright-report/
|
||||
test-results/
|
||||
|
||||
/otche
|
||||
bin/
|
||||
*.test
|
||||
coverage.out
|
||||
*.coverprofile
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
FROM golang:1.26-bookworm AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY cmd ./cmd
|
||||
COPY internal ./internal
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/otche ./cmd/otche
|
||||
FROM debian:trixie-slim AS api
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && rm -rf /var/lib/apt/lists/* && groupadd -g 10001 otche && useradd -u 10001 -g otche -M otche
|
||||
COPY --from=build /out/otche /usr/local/bin/otche
|
||||
USER 10001:10001
|
||||
WORKDIR /var/lib/otche
|
||||
ENTRYPOINT ["otche"]
|
||||
CMD ["api"]
|
||||
|
||||
FROM api AS worker
|
||||
USER root
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg xorriso && rm -rf /var/lib/apt/lists/*
|
||||
COPY windows /opt/otche/windows
|
||||
USER 10001:10001
|
||||
CMD ["worker"]
|
||||
@@ -0,0 +1,36 @@
|
||||
# Otche backend
|
||||
|
||||
Go API, worker and watchdog, Windows PowerShell runner, PostgreSQL schema, RFB/H.264 recorder and reusable trusted operator tools for the Otche Windows/Defender observation service. Results are observations, not a certificate that an uploaded file is safe.
|
||||
|
||||
## Repository ownership
|
||||
|
||||
This is the canonical backend source. API, worker and watchdog are **separate processes/containers built from one Go module and one command**; their shared internal packages are not duplicated into services. The unsigned reviewed `windows/` source is owned here; operators sign their own deployment copy. Frontend source belongs only to [otche-frontend](https://git.qomar.pw/otche/otche-frontend). Compose and portable installation instructions belong to [otche-deploy](https://git.qomar.pw/otche/otche-deploy). Clone the three repositories as siblings; there is no parent monorepo or submodule requirement.
|
||||
|
||||
## Build
|
||||
|
||||
```sh
|
||||
go mod download
|
||||
go build -trimpath -o ./build/otche ./cmd/otche
|
||||
docker build --target api -t otche-api:local .
|
||||
docker build --target worker -t otche-worker:local .
|
||||
```
|
||||
|
||||
Use Go 1.26 (the module minimum is in `go.mod`). Docker builds use Go 1.26/Bookworm and Debian Trixie runtime; worker adds FFmpeg and xorriso. `internal/store/schema.sql` is embedded in the binary. Runtime user is UID/GID 10001. No private key or operator credential is included.
|
||||
|
||||
The command modes are `api`, `worker`, `watchdog`, `migrate`, `user-create`, `bindings-sync`, `source-maintenance` and `source-validate`. See [configuration](docs/CONFIG.md) and [API contract](docs/API.md). `DATABASE_URL_FILE` is required for all database commands. API additionally uses `PUBLIC_ORIGIN`, `ARTIFACT_ROOT`, optional `LISTEN_ADDR` (default :8080), and explicitly opt-in `ALLOW_INSECURE_HTTP=true` only for loopback development. Run `migrate` before the API. Use the sibling deploy repository for a complete database-backed startup and secure first-user creation.
|
||||
|
||||
## Verification
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Database-backed tests require a **dedicated disposable PostgreSQL database** via `OTCHE_TEST_DATABASE_URL`, never a production DSN. Recorder tests use FFmpeg. Inspect test prerequisites before interpreting skips as success. Windows helper tests are `windows/Test-OtcheArguments.ps1` and `windows/Test-OtcheGrub.ps1`; run only under the existing approved execution policy. They do not qualify a source image. No automatic source VM operations are part of a normal build or test.
|
||||
|
||||
## Execution prerequisites and safety
|
||||
|
||||
Read [Windows setup and runner contract](windows/Source-Setup.txt) before preparing a dedicated stopped ordinary Windows VM. Review and sign every PS1/PSM1 on an authorized signing workstation; provide trusted certificates and credentials separately. The project does not bypass execution policy, disable Defender/UAC/Secure Boot, or install private signing keys on guests. Samples run only on disposable full clones after isolation and source qualification. Grub collects bounded literal local files under the selected interactive token, not as arbitrary SYSTEM/host reads.
|
||||
|
||||
`provisioning/onboard-owner.py` is an explicit trusted administrator tool (Python 3.9+, PVE CLI on an approved node). Dry-run is default; applying creates scoped credentials and refuses collisions. `probe-owner-isolation.py` reads operator-supplied disposable probes and writes private short-lived evidence; it never invents successful proofs. `install-extractor.sh` and `otche-extract` are a matched pair for an independently approved clean Debian extractor guest, **not** the control host. Optional online/extractor execution remains blocked without genuine matching proofs.
|
||||
|
||||
Protected VMIDs 7000/7001 and forbidden shared storage names in the code are intentional safety restrictions, not deployment inventory. Retain them. Choose actual owner resources explicitly and do not remove protections to make a configuration pass. No live host inventory, sample payload, proof, certificate, credentials or historical acceptance log is distributed here.
|
||||
@@ -0,0 +1,133 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"otche/internal/api"
|
||||
"otche/internal/store"
|
||||
"otche/internal/worker"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
slog.Error("Otche stopped", "error", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
func env(name, fallback string) string {
|
||||
if v := os.Getenv(name); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
func run() error {
|
||||
if len(os.Args) < 2 {
|
||||
return errors.New("usage: otche api|worker|watchdog|migrate|user-create|source-maintenance|source-validate|bindings-sync")
|
||||
}
|
||||
mode := os.Args[1]
|
||||
flags := flag.NewFlagSet(mode, flag.ContinueOnError)
|
||||
username := flags.String("username", "", "Local Otche username")
|
||||
role := flags.String("role", "operator", "admin or operator")
|
||||
passwordStdin := flags.Bool("password-stdin", false, "Read password securely from stdin, not command arguments")
|
||||
passwordFile := flags.String("password-file", "", "Read password from an operator-protected file")
|
||||
profile := flags.String("profile-id", "", "Existing profile UUID")
|
||||
source := flags.String("source-ref", "", "Configured worker source reference")
|
||||
if err := flags.Parse(os.Args[2:]); err != nil {
|
||||
return err
|
||||
}
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
url, err := store.Secret("DATABASE_URL")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
db, err := store.Open(ctx, url)
|
||||
if err != nil {
|
||||
return errors.New("database connection failed; check DATABASE_URL secret and server")
|
||||
}
|
||||
defer db.Close()
|
||||
switch mode {
|
||||
case "migrate":
|
||||
return store.Migrate(ctx, db)
|
||||
case "user-create":
|
||||
if *passwordStdin == (*passwordFile != "") {
|
||||
return errors.New("choose exactly one of --password-stdin or --password-file")
|
||||
}
|
||||
var b []byte
|
||||
if *passwordStdin {
|
||||
b, err = io.ReadAll(io.LimitReader(os.Stdin, 1024))
|
||||
} else {
|
||||
b, err = os.ReadFile(*passwordFile)
|
||||
}
|
||||
if err != nil {
|
||||
return errors.New("could not read password input")
|
||||
}
|
||||
password := strings.TrimRight(string(b), "\r\n")
|
||||
id, e := api.CreateUser(ctx, db, *username, password, *role)
|
||||
for i := range b {
|
||||
b[i] = 0
|
||||
}
|
||||
if e != nil {
|
||||
return errors.New("user creation failed: use unique username, admin/operator role and 14-72 byte password")
|
||||
}
|
||||
fmt.Println(id)
|
||||
return nil
|
||||
case "api":
|
||||
maxUpload, _ := strconv.ParseInt(env("MAX_UPLOAD_BYTES", "536870912"), 10, 64)
|
||||
maxOwner, _ := strconv.ParseInt(env("MAX_OWNER_BYTES", "8589934592"), 10, 64)
|
||||
maxQueue, _ := strconv.Atoi(env("MAX_QUEUED_JOBS", "20"))
|
||||
handler, e := api.New(db, api.Config{Origin: env("PUBLIC_ORIGIN", "http://localhost:8080"), ArtifactRoot: env("ARTIFACT_ROOT", "./artifacts"), SecureCookies: env("ALLOW_INSECURE_HTTP", "false") != "true", MaxUploadBytes: maxUpload, MaxOwnerBytes: maxOwner, MaxQueuedJobs: maxQueue})
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
server := &http.Server{Addr: env("LISTEN_ADDR", ":8080"), Handler: handler, ReadHeaderTimeout: 10 * time.Second, ReadTimeout: 10 * time.Minute, WriteTimeout: 10 * time.Minute, IdleTimeout: 60 * time.Second, MaxHeaderBytes: 16 << 10}
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- server.ListenAndServe() }()
|
||||
select {
|
||||
case e = <-done:
|
||||
if errors.Is(e, http.ErrServerClosed) {
|
||||
return nil
|
||||
}
|
||||
return e
|
||||
case <-ctx.Done():
|
||||
shutdown, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
return server.Shutdown(shutdown)
|
||||
}
|
||||
case "worker":
|
||||
return worker.Run(ctx, db, env("ARTIFACT_ROOT", "./artifacts"), env("WORKER_CONFIG_FILE", "/run/secrets/worker-config.json"))
|
||||
case "watchdog":
|
||||
return worker.Watchdog(ctx, db, env("WORKER_CONFIG_FILE", "/run/secrets/worker-config.json"))
|
||||
case "bindings-sync":
|
||||
return worker.SyncBindings(ctx, db, env("WORKER_CONFIG_FILE", "/run/secrets/worker-config.json"))
|
||||
case "source-maintenance":
|
||||
if *source == "" {
|
||||
return errors.New("--source-ref required")
|
||||
}
|
||||
return worker.BeginMaintenance(ctx, db, env("WORKER_CONFIG_FILE", "/run/secrets/worker-config.json"), *source)
|
||||
case "source-validate":
|
||||
if *profile == "" || *source == "" {
|
||||
return errors.New("--profile-id and --source-ref required")
|
||||
}
|
||||
id, e := worker.ValidateSource(ctx, db, env("WORKER_CONFIG_FILE", "/run/secrets/worker-config.json"), *profile, *source)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
fmt.Println(id)
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("unknown mode %q", mode)
|
||||
}
|
||||
}
|
||||
+116
@@ -0,0 +1,116 @@
|
||||
# Отче HTTP API contract
|
||||
|
||||
Base `/api/v1`; JSON snake_case; RFC3339 UTC dates; UUID IDs; errors `{ "error": { "code": "invalid_request", "message": "..." } }`. Lists `{ "items": [...] }`. Same origin only; browser fetch `credentials: include`. Session cookie HttpOnly SameSite=Strict Secure (local development flag permits HTTP). `GET /auth/session` returns `{user:{id,username,role},csrf_token}` or 401; `POST /auth/login` `{username,password}` returns same shape and cookie. All authenticated writes require `X-CSRF-Token` plus matching Origin. `POST /auth/logout` returns 204. Roles `admin|operator`. Admin reads all jobs; operators only own jobs. No browser PVE secrets.
|
||||
|
||||
## Job submission and settings
|
||||
|
||||
`POST /uploads` body raw bytes (`application/octet-stream`), filename in `X-Filename` (encodeURIComponent). Response 201 `{id,filename,size,sha256,created_at}`. Original bytes immutable. No multipart. `GET /uploads/{id}/content` authorized attachment.
|
||||
|
||||
`POST /jobs` `{upload_id,profile_ids:[UUID],settings:{internet:"offline"|"online",duration_seconds:30|60|90|120|180|300|600|900|1200,filename:"original"|"random",privilege:"user"|"admin",args_mode:"none"|"custom",args: string[],set_zoneid:boolean,dll_mode:"regsvr32"|"rundll32",dll_export:string,architecture:"auto"|"x86"|"x64",wsh_host:"cscript"|"wscript",msi_ui:"full"|"quiet"|"passive"}}` -> 201 Job. Arguments are an explicit JSON string array; only `{sample.path}` and `{sample.name}` expand in guest after path selection. DLL export mandatory with rundll32, never guessed. MSI logs always collected. Defender only; each selected profile creates exactly one Run. Immutable settings, random basename selected once per Job preserving extension.
|
||||
|
||||
Optional immutable `settings.grub_paths: string[]` adds **Grub** file collection after the observation window, on the selected user/admin token. Omitted/empty means no archive and remains valid for historical Jobs. At most 32 literal absolute local Windows file paths, 1024 UTF-8 bytes each / 8192 aggregate. No environment or `{sample.*}` expansion: percent/braces inside an absolute filename are literal. UNC/device/network paths, ADS, traversal, wildcards and unsafe components are rejected; actual directories, reparse points and hard links are rejected in the guest. Extensionless ordinary files are allowed. Serialized settings (including escaped arguments) must fit 38 KiB; the full worker manifest must fit the existing 40 KiB transport limit before allocation.
|
||||
|
||||
Each Attempt with Grub produces one private `kind: "grub_archive"`, `filename: "grub.zip"`, `content_type: "application/zip"` artifact through the existing authorized attachment URL. ZIP entries are controller-generated `files/001-sanitized_basename` etc.; duplicate basenames remain distinct. `manifest.json` contains the validated Grub report. Missing/inaccessible/oversize files are explicit errors, not Defender findings; an all-missing request produces an honest manifest-only `empty` archive. Per-file errors alone do not retain clones when ordinary evidence is complete. Transport/hash/archive/publication failure retains stopped evidence; ZIP and report metadata commit before clone deletion.
|
||||
|
||||
Readable live logs are captured as bounded **best-effort open-length snapshots**, not atomic filesystem snapshots. Writers that allow reads are supported; growth is not chased. Detected length/content/write-time changes yield `changed`, preserve the captured bytes when possible and make the archive `partial`. A sharing-denied file may have no snapshot. Limits: per file `min(8 MiB,max_artifact_bytes/2)`, total captured bytes `min(32 MiB,max_artifact_bytes/2)`; existing collection deadline and owner quota apply. No guest-supplied ZIP is trusted or imported.
|
||||
|
||||
`GET /jobs?page=1&page_size=25&q=&status=` -> `{items:Job[],total,page,page_size,next_page:null|number}`; `GET /jobs/{id}` Job with runs; `POST /jobs/{id}/cancel` -> Job; `POST /jobs/{id}/retry` -> same Job with a new Attempt for each terminal Run, preserving every prior Attempt and original immutable settings/revisions, admission rechecked. Active Jobs cannot retry. Job `{id,owner_id,upload_id,filename,execution_filename,sha256,size,settings,status,created_at,updated_at,cancel_requested,runs:Run[]}`. Status `queued|running|completed|cancelled|failed`.
|
||||
|
||||
Run `{id,job_id,profile_id,profile_name,os_build:string|null,revision_id,antivirus:"defender",status,attempts:Attempt[]}`; status `queued|running|completed|cancelled|failed`. `os_build` belongs to the Run's exact revision, never the profile's later current revision.
|
||||
Attempt `{id,run_id,command_id,phase,outcome,findings,telemetry,cleanup,error,created_at,started_at,finished_at,deadline_at,allocation,report,artifacts:Artifact[]}`.
|
||||
|
||||
- phase: `queued|provisioning|booting|recording|delivering|preparing|running|collecting|stopping|cleanup|finished`.
|
||||
- outcome: `pending|executed|blocked_before_execution|incompatible|policy_blocked|delivery_error|interrupted|cancelled|error`.
|
||||
- findings: `unknown|detected|not_observed`; telemetry: `pending|complete|partial|unavailable`; cleanup: `pending|complete|evidence_held|failed`.
|
||||
- Timestamps are null when not applicable. Early quarantine has no actual start/PID/duration; unknown session is null, **not Session 0**.
|
||||
- allocation is null for operators; admins receive `{id,node,vmid,state}` diagnostics only, never PVE URLs or credentials.
|
||||
- report is null before collection. Partially recovered data may have null environment/Defender state; UI must show N/A, not infer successful execution.
|
||||
|
||||
```typescript
|
||||
type ExecutionReport = {
|
||||
execution: {
|
||||
exit_code: number | null;
|
||||
error: string;
|
||||
actual_duration_seconds: number | null;
|
||||
started_at: string | null;
|
||||
finished_at: string | null;
|
||||
user: string;
|
||||
session_id: number | null;
|
||||
pid: number | null;
|
||||
path: string;
|
||||
arguments: string[];
|
||||
handler: string;
|
||||
privilege: string;
|
||||
};
|
||||
defender: {
|
||||
before: DefenderState | null;
|
||||
after: DefenderState | null;
|
||||
drift: boolean;
|
||||
detections: Detection[];
|
||||
};
|
||||
environment: {
|
||||
os_build: string;
|
||||
architecture: string;
|
||||
powershell_version: string;
|
||||
execution_policy: string;
|
||||
runner_version: string;
|
||||
} | null;
|
||||
collection_errors: string[];
|
||||
grub?: {
|
||||
status: 'complete' | 'partial' | 'empty';
|
||||
requested: number;
|
||||
collected: number; // archived files, including explicitly changed snapshots
|
||||
files: Array<{
|
||||
requested_path: string;
|
||||
resolved_path: string | null;
|
||||
member: string | null;
|
||||
status: 'collected' | 'missing' | 'access_denied' | 'invalid_path' | 'changed' | 'oversize' | 'error';
|
||||
error: string;
|
||||
size: number | null; // zero is a real captured empty file, never unknown
|
||||
sha256: string | null;
|
||||
snapshot: { open_size: number; changed: boolean; consistency: 'best_effort' } | null;
|
||||
}>;
|
||||
};
|
||||
};
|
||||
type DefenderState = {
|
||||
active: boolean;
|
||||
platform: string;
|
||||
engine: string;
|
||||
intelligence: string;
|
||||
intelligence_updated_at: string;
|
||||
fingerprint: string;
|
||||
preferences: Record<string, unknown>;
|
||||
};
|
||||
type Detection = {
|
||||
name: string;
|
||||
id: string;
|
||||
action: string;
|
||||
resources: string[];
|
||||
timestamp: string;
|
||||
stage: 'delivery' | 'preparation' | 'execution' | 'collection';
|
||||
source: 'defender' | 'smartscreen' | 'policy';
|
||||
};
|
||||
```
|
||||
|
||||
Missing or partial telemetry is not an empty clean report. Fixed collector fields, not arbitrary raw guest objects, populate the DTO.
|
||||
|
||||
`GET /jobs/{id}/events?after=N` -> `{items:[{id,job_id,attempt_id,kind,message,created_at}]}`; polling supported, no PVE WebSocket exposed. `GET /jobs/{id}/artifacts` -> list Artifact `{id,job_id,attempt_id,kind,filename,content_type,size,sha256,created_at,url}`. `GET /artifacts/{id}/content` owner-authorized, supports video Range, forces attachment except safe video. HTML always attachment + sandbox. `GET /attempts/{id}/video` -> `{state:"pending"|"recording"|"complete"|"partial"|"unavailable",segments:Artifact[],gaps:[{at,reason}],started_at,finished_at}`.
|
||||
|
||||
## Profiles and administration
|
||||
|
||||
`GET /profiles` -> list Profile `{id,name,os,os_build:string|null,architecture,enabled,qualification,state,current_revision_id,online_available,reason,metadata:object}`; state `maintenance|published`; qualification `unqualified|qualified|drifted`; metadata contains observed build/Defender baseline/policy only, no VM or token configuration to operators. Admission rejects unqualified/unconfigured profiles, not simulated runs.
|
||||
|
||||
Windows display labels preserve the user-defined base `name` and append the verified `os_build` (`CurrentBuildNumber.UBR`, e.g. `26200.6584`). Null is explicitly shown as `build не определён`; release labels such as 25H2 are not a build source. The field is derived only from successful qualification evidence for the exact revision: the original captured `environment.os_build` remains unchanged for fingerprints, while only its first two numeric components are projected for display. Historical qualification records without embedded environment use their own recorded qualification-control Attempt report, not arbitrary sample reports or current profile metadata. Profile lists, admin revision lists and Job/Run list/detail projections expose this consistently in bounded single SQL queries; historical Job settings/revision IDs are not rewritten.
|
||||
|
||||
Admin: `GET /admin/users`; `POST /admin/users` `{username,password,role}`; `PATCH /admin/users/{id}` `{role?,disabled?,password?}`. User `{id,username,role,disabled,created_at}`.
|
||||
`GET /admin/profiles`; `POST /admin/profiles` `{name,os,architecture}` -> Profile (maintenance, unqualified). `PATCH /admin/profiles/{id}` `{name?,enabled?}`. `POST /admin/profiles/{id}/maintenance` closes admission, returns profile plus drain status; never powers off live master. `POST /admin/profiles/{id}/publish` `{revision_id}` selects a previously worker-validated stopped source revision; never accepts VMID from browser. `POST /admin/profiles/{id}/qualify` queues qualification, returns `{id,status}`. Qualified revision configuration and credentials are worker-only CLI/file-backed, not browser secrets.
|
||||
`GET /admin/bindings` -> list `{owner_id,pool,iso_storage,disk_storage,node,configured,reason,isolation_expires_at}` (metadata only). `isolation_expires_at` is nullable RFC3339: null means not validated; effective `configured` requires successful worker validation **and** a future expiry at request time. Expired/missing expiry closes dashboard readiness and `POST /jobs` admission even if a stale stored flag was true. Bindings are provisioned by operator CLI with file-backed credentials and genuine probe evidence, no secret edits/browser storage; UI shows actual expiry/prerequisites rather than a fake credential form.
|
||||
`GET /admin/health` -> `{database,worker,last_worker_seen,integration_ready,blockers:string[]}`.
|
||||
`POST /admin/attempts/{id}/release-evidence` `{confirm:true}` explicit irreversible authorized cleanup of owned retained disposable VM/disk only, never master/control.
|
||||
`GET /dashboard` owner-scoped `{metrics:{jobs,queued,running,completed,failed,cancelled,detected},recent_jobs:Job[],queue:{queued,running},integration:{ready:boolean,blockers:string[]}}`.
|
||||
`GET /admin/profiles/{id}/revisions` -> `{items:[{id,profile_id,fingerprint,config_digest,qualification,created_at}]}`. Revision `qualification` is null or an object with `worker_validated:boolean`, `status:"passed"|"failed"`, `state:"qualified"|"unqualified"`, `baseline_fingerprint:string` and observed control details; fields absent before that stage must remain unknown. It is not the profile's string qualification enum.
|
||||
`GET /admin/profiles/{id}/qualifications` -> `{items:[{id,profile_id,status,result,created_at}]}`; `GET /admin/qualifications/{id}` same qualification object. Status `queued|running|passed|failed`; result actual controls/attempt IDs/errors only, null until available. Qualification bypasses *qualified-profile* admission only; still requires explicit configured stopped source and isolated disposable allocation, never puts controls into master or a real Run.
|
||||
|
||||
## Deployment
|
||||
|
||||
The sibling frontend Vite build writes `dist/`; nginx reverse proxy same-origin `/api/` to Go API:8080, static UI fallback. Go API and worker separate containers/process modes. PostgreSQL, private artifact volume; PVE secret config mounted in worker only. No demo data/runtime mock adapters. Empty installs show real empty states and fail-closed integration prerequisites.
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
# Worker configuration and qualification proofs
|
||||
|
||||
Use `provisioning/runtime-config.example.json` as the complete base; replace metadata and paths, never copy an invented `passed:true` proof. Runtime reads this file only in worker/watchdog. All token/CA/proof paths are absolute inside their containers; seals belong in writable `/var/lib/otche/source-seals/`, not read-only `/run/otche`.
|
||||
|
||||
## Credentials
|
||||
|
||||
Each owner's five distinct files contain `{ "token_id": "service@pve!worker", "secret": "secret-manager-value" }`. Tokens are scoped separately; do not commit these files. Each owner has its own pool and private ISO storage. Distinct names backed by the same directory are not isolation. User/token effective permissions are intersected. `provisioning/onboard-owner.py --help` describes repeated `--source-vmid` and optional `--extractor-source-vmid`; dry-run is default. Use `--apply` only on reviewed private resources.
|
||||
|
||||
The supported single-account deployment uses existing `otche@pve` (firstname/comment `otche`, no password) with five separate `privsep=1` tokens. Pass `--service-user otche@pve` to trusted onboarding. The account must already be enabled, non-expiring, and have no groups, tokens, ACL entries or effective resource privileges; an existing privilege-bearing account is refused, not reset. Token names include the owner prefix and purpose. The parent user receives only the union of the reviewed per-token grants on explicit paths; each token receives only its purpose's subset. No ACL is granted on `/`, `/vms`, `/storage`, or `/access`. Keep the account password unset: a password login would expose the parent union rather than a single token's subset.
|
||||
|
||||
The control container calls `https://pve.example.invalid:8006/api2/json` using `Authorization: PVEAPIToken=<token_id>=<secret>`, with the cluster CA and certificate IP/hostname verification. This is an outbound HTTPS call, not a PVE password login, guest SSH, or mounted host filesystem. Only the worker/watchdog credential mount receives these files. The API, frontend, browser and Windows guest receive neither token nor PVE configuration; do not mount the worker directory into those services. Store credential files mode `0600`, parent directories `0700`, readable only by the worker UID and the deployment administrator; mount them read-only.
|
||||
|
||||
Onboarding refuses pool/role/user collisions and overlapping directory storage before writing credentials. After a partial apply, inspect the exact scoped resources rather than blindly repeating or resetting the user. Five file paths do not by themselves prove separation: confirm all five token IDs differ, `privsep=1`, and inspect each token's effective privileges through the actual verified-TLS CT-to-PVE transport. Read-only authentication evidence is not source/Windows qualification and must not be converted into a passed isolation proof.
|
||||
|
||||
`bindings-sync` authenticates **all five** credentials through the application's Go PVE client (`GET /access/permissions`) before examining isolation evidence or setting `configured=true`. An invalid/revoked runtime, recorder, uploader or housekeeping token blocks the binding even when provisioner works and files exist. Successful authentication alone does not open admission: missing/expired isolation proof, unavailable private storage and source qualification remain separate fail-closed gates.
|
||||
|
||||
PVE token-management safety must be checked independently of privilege separation; do not grant User.Modify or broad parent-user privileges. Authentication alone is not isolation or source qualification.
|
||||
|
||||
Directory image volumes are supported as `storage:VMID/vm-VMID-disk-N.qcow2` (also raw/vmdk); both directory and filename must match the allocated VMID. ZFS/LVM-style `storage:vm-VMID-disk-N` remains supported. Source/base volumes, mismatched ownership and traversal are rejected. Validate these restrictions on disposable resources before enabling execution.
|
||||
|
||||
Job ISO labels use `OT` plus the first 14 uppercase hexadecimal digits of the job UUID: exactly 16 ASCII characters, preserved identically in both ISO9660 primary and Joliet supplementary volume descriptors and Windows `VolumeLabel`. The label only locates candidate media; `job.json` must still match the complete job UUID, original SHA-256 and filename. Do not shorten those identity checks to the label prefix.
|
||||
|
||||
Python 3.13 enables an optional X509_STRICT profile that rejects the older PVE cluster CA because it lacks a CA keyUsage extension. The isolation probe's `verified_context` explicitly omits only that optional strict profile, matching the Go/curl validation contract: `CERT_REQUIRED`, trusted chain, certificate dates and hostname checks remain enabled. Verify wrong CA and wrong hostname rejection against your own deployment. No `-k`, `CERT_NONE`, `check_hostname=false`, changed cluster CA or swallowed TLS failure is used. Curl Authorization must be provided through stdin/private config, never a secret-bearing `-H` argument.
|
||||
|
||||
`provisioning/probe-owner-isolation.py` performs real **read-only** allow/deny checks, refuses 404/500 as proof of denial, checks exact effective privilege key allowlists (permission values describe propagation, not whether granted), and requires trusted nonalias storage inventory. Example:
|
||||
|
||||
```sh
|
||||
python3 provisioning/probe-owner-isolation.py \
|
||||
--config /secure/config.json \
|
||||
--owner 11111111-1111-4111-8111-111111111111 \
|
||||
--foreign-owner 22222222-2222-4222-8222-222222222222 \
|
||||
--own-disposable 8101 --foreign-disposable 8102 --source-vmid 7001 \
|
||||
--storage-inventory /secure/storage-inventory.json \
|
||||
--evidence /secure/owner-one-isolation-evidence.json \
|
||||
--proof /secure/owner-one-isolation-proof.json
|
||||
```
|
||||
|
||||
8101/8102 are **examples only**: use actual independently created disposable probes, never production guests. Administrator storage inventory is obtained through the PVE read API `/storage` and protected locally; symlink/dataset aliases require independent operator inspection. The tool does not create probes, execute payloads, modify firewall or mutate VMs. Destructive deny tests are never run on production. Copy completed proof into the corresponding configured `isolation_proof_file` without changing its bytes. One-day expiry requires revalidation, not editing the date.
|
||||
|
||||
Proofs are deployment-specific, expire after one day and require real revalidation. No acceptance logs or pre-passed proofs ship with this repository.
|
||||
|
||||
Binding metadata exposes `isolation_expires_at` (RFC3339 or null). API admission, dashboard and admin binding status evaluate expiry at request time; the worker also clears stale stored readiness. Null is unvalidated, and expired proof requires fresh real probes, never an edited expiry date. Source qualification is a separate required step on disposable clones. No published deployment state is implied by the example configuration.
|
||||
|
||||
Readiness now uses the signed fixed `Otche-DesktopReady` Limited/Interactive task, with a fresh nonce, bounded response and deadline, matching boot/account/console session, unlocked Default input desktop, real Explorer shell/taskbar and no visible setup/sign-in host. Resident UserOOBEBroker or generic WWAHost processes alone are not blockers. During a sample, the worker uses `-BootOnly`; post-run it uses `-BaselineOnly`, so sample-owned GUI is not reclassified as setup and no repeated interactive probe is launched. Capture and review your own protected source/fresh-clone evidence. Observed Defender and Windows versions are recorded, never inferred from marketing release names.
|
||||
|
||||
## Optional online policy
|
||||
|
||||
Under the owner binding set:
|
||||
|
||||
```json
|
||||
{
|
||||
"online": {
|
||||
"bridge": "otche-isolated",
|
||||
"proof_file": "/run/otche/owner-one/online-proof.json"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The private online proof object has fields:
|
||||
|
||||
- `owner_id`, `node`, `bridge`: exact binding identities.
|
||||
- `passed`, `management_denied`, `private_networks_denied`, `cloud_consent`: actual operator-qualified boolean results/approval, all required true.
|
||||
- `expires_at`: RFC3339 UTC expiry.
|
||||
- `network_sha256`: SHA-256 of canonical JSON (Go encoding/json map-key order) of the actual PVE node network/bridge object, with `digest` removed.
|
||||
- `firewall_options_sha256`: canonical JSON SHA-256 of VM firewall options without `digest`.
|
||||
- `firewall_rules_sha256`: canonical JSON SHA-256 of VM firewall rules with per-row `digest` removed.
|
||||
- `evidence_sha256`: SHA-256 of protected actual network probe evidence.
|
||||
|
||||
This is an operator-produced approval artifact, **not an automatic network deployment**. Capture real management/private-network denied egress tests from a disposable clone and approved public connectivity; independently ensure external firewall is active. VM policy_in/policy_out must DROP; exact approved rules permit required online traffic. Source inherits the one approved isolated NIC; clone enables only that NIC with firewall flag. Worker compares live network/rule/options objects and refuses drift. Never alter a cluster firewall automatically to satisfy this prerequisite. Online qualification additionally runs its own EICAR/benign controls; a proof file alone does not mark a profile online_available.
|
||||
|
||||
Online fingerprints are bound to actual API responses and installed PVE semantics; do not fabricate example hashes. Online network qualification must be performed separately by the deployment operator. Successful PVE token authentication does not establish guest egress isolation; this gate stays closed until valid operator artifacts exist.
|
||||
|
||||
## Optional read-only extractor
|
||||
|
||||
Top-level configuration:
|
||||
|
||||
```json
|
||||
{
|
||||
"extractor": {
|
||||
"node": "pve-node",
|
||||
"source_vmid": 7101,
|
||||
"config_digest": "ACTUAL_PVE_CONFIG_DIGEST",
|
||||
"proof_file": "/run/otche/extractor-proof.json",
|
||||
"source_disk": "scsi0",
|
||||
"max_disk_bytes": 17179869184,
|
||||
"include_crash_dump": true,
|
||||
"timeout_seconds": 180
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
7101 is an example, not a discovered VM. `source_disk` is the owned Windows disk to extract (default scsi0, preserving system evidence); `max_disk_bytes` is reserved space for the full Linux extractor clone. `timeout_seconds` must 30–300. Source is stopped ordinary Linux, no NIC/USB/PCI passthrough, scsi1 empty, QGA installed, fixed `otche-extract` installed through `install-extractor.sh`. No disk is formatted by these tools.
|
||||
|
||||
Extractor proof fields: `passed`, `node`, `source_vmid`, `config_digest`, `expires_at`, `evidence_sha256`, `read_only_verified`, `no_egress`. Require actual read-only reassignment verification (PVE ro flag, Linux blockdev RO, attempted writes denied on benign disposable evidence), exact serial+size, no egress, bounded QGA export. The runtime does not invent or rewrite this proof.
|
||||
|
||||
The worker journals source/destination ownership, task IDs and disk moves in `extractor_allocations`. Before boot, borrowed evidence is `ro=1` and gets an allocation-bound serial. No pending config is accepted. Original Windows VM and its disks remain retained on collection failure, including missing config, encryption, dirty filesystem, timeout and ambiguous UPID. The Linux VM uses ntfs-3g `ro,norecover`; neither PVE nor control CT mounts NTFS. It stops externally on completion/failure. Evidence must return read-only to the original stopped Windows VM before extractor deletion; unresolved states remain journaled for watchdog/operator.
|
||||
|
||||
## Enable execution only after configuration
|
||||
|
||||
```sh
|
||||
docker compose --profile execution run --rm worker bindings-sync
|
||||
docker compose --profile execution run --rm worker source-maintenance --source-ref win11-stopped-revision
|
||||
# Operator performs approved source maintenance and graceful shutdown separately.
|
||||
docker compose --profile execution run --rm worker source-validate \
|
||||
--profile-id ACTUAL_PROFILE_UUID --source-ref win11-stopped-revision
|
||||
docker compose --profile execution up -d worker watchdog
|
||||
```
|
||||
|
||||
After source validation, admin queues qualification, reviews actual evidence, and publishes only a passed immutable revision. Missing account credentials/script policy/signing trust, stale Defender or inaccessible QGA remain actionable prerequisites. Never bypass protections or change a queued source revision to make admission pass.
|
||||
|
||||
## Grub artifact collection
|
||||
|
||||
`settings.grub_paths` is optional immutable Job data, not a worker host path. The interactive runner captures literal local files under the already-selected user/admin token after observation; the SYSTEM telemetry collector exports **only fixed numbered staged files** into the protected control directory after no-reparse/hardlink and hash/size validation. It never opens requested Grub paths as SYSTEM. Requested paths are never resolved on the CT, PVE host or workstation, nor interpolated into shell source.
|
||||
|
||||
Grub reserves one additional `max_artifact_bytes` bucket in the existing owner artifact quota and headroom check. Payload limits are `min(8 MiB,max_artifact_bytes/2)` per file and `min(32 MiB,max_artifact_bytes/2)` total. Snapshot and protected staging passes each use at most half `collect_timeout_seconds`; the worker still applies its existing collection/safety deadlines. Staged files are not individually published: the controller streams one ZIP, hashes transported bytes, writes a controller-owned manifest and atomically publishes ZIP metadata plus validated report under the current attempt lease. A unique exclusive private archive path prevents replacing earlier immutable evidence. Ambiguous commit preserves bytes.
|
||||
|
||||
Ordinary per-file errors and best-effort changed snapshots make Grub `partial`/`empty`, independently of Defender verdict and telemetry. They do not by themselves retain clones. Missing export, corruption, timeout or private persistence failure does retain stopped evidence. No Grub archive is created for historical jobs without paths. Guest administrator tampering remains within the documented untrusted guest-evidence boundary; Grub files must never be treated as trusted code.
|
||||
|
||||
## Windows build identity
|
||||
|
||||
The signed baseline already reads actual `CurrentBuildNumber`, `UBR` and `BuildLabEx` into `environment.os_build`; no release-to-build lookup or manual VM-name rename is used. Qualification persists that observed environment on its exact revision. API/UI build labels project only `CurrentBuildNumber.UBR`, preserving the full original fingerprinted string. Older successful revisions resolve through their recorded qualification control attempts for the same revision. Unknown/unqualified builds remain null, and a historical Run never borrows the current profile's build after maintenance or publication of another revision.
|
||||
@@ -0,0 +1,17 @@
|
||||
module otche
|
||||
|
||||
go 1.24.0
|
||||
|
||||
require (
|
||||
github.com/gorilla/websocket v1.5.3
|
||||
github.com/jackc/pgx/v5 v5.7.5
|
||||
golang.org/x/crypto v0.38.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
golang.org/x/sync v0.14.0 // indirect
|
||||
golang.org/x/text v0.25.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,23 @@
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
|
||||
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.7.5 h1:JHGfMnQY+IEtGM63d+NGMjoRpysB2JBwDr5fsngwmJs=
|
||||
github.com/jackc/pgx/v5 v5.7.5/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
golang.org/x/crypto v0.38.0 h1:jt+WWG8IZlBnVbomuhg2Mdq0+BBQaHbtqHEFEigjUV8=
|
||||
golang.org/x/crypto v0.38.0/go.mod h1:MvrbAqul58NNYPKnOra203SB9vpuZW0e+RRZV+Ggqjw=
|
||||
golang.org/x/sync v0.14.0 h1:woo0S4Yywslg6hp4eUFjTVOyKt0RookbpAHG4c1HmhQ=
|
||||
golang.org/x/sync v0.14.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
|
||||
golang.org/x/text v0.25.0 h1:qVyWApTSYLk/drJRO5mDlNYskwQznZmkpV2c8q9zls4=
|
||||
golang.org/x/text v0.25.0/go.mod h1:WEdwpYrmk1qmdHvhkSTNPm3app7v4rsT8F2UD6+VHIA=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,344 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/crypto/bcrypt"
|
||||
"otche/internal/store"
|
||||
)
|
||||
|
||||
// Build labels come only from successful qualification of this exact revision.
|
||||
// Older revisions retain their recorded control-attempt evidence; no current
|
||||
// profile metadata or arbitrary sample report may rewrite a historical build.
|
||||
const revisionOSBuild = `CASE WHEN v.qualification->>'status'='passed' THEN substring(COALESCE(v.qualification#>>'{environment,os_build}',(SELECT a.report#>>'{environment,os_build}' FROM jsonb_array_elements(CASE WHEN jsonb_typeof(v.qualification->'controls')='array' THEN v.qualification->'controls' ELSE '[]'::jsonb END) WITH ORDINALITY c(control,ordinal) JOIN attempts a ON a.id=CASE WHEN c.control->>'attempt_id' ~ '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$' THEN (c.control->>'attempt_id')::uuid ELSE NULL END JOIN runs qr ON qr.id=a.run_id WHERE qr.revision_id=v.id AND c.ordinal<=4 AND a.report#>>'{environment,os_build}' IS NOT NULL ORDER BY c.ordinal LIMIT 1)) FROM '^([0-9]+\.[0-9]+)(\.|$)') ELSE NULL END`
|
||||
const profileProjection = `to_jsonb(p)||jsonb_build_object('os_build',(SELECT ` + revisionOSBuild + ` FROM revisions v WHERE v.id=p.current_revision_id))`
|
||||
const runProjection = `to_jsonb(r)||jsonb_build_object('profile_name',p.name,'os_build',(SELECT ` + revisionOSBuild + ` FROM revisions v WHERE v.id=r.revision_id))`
|
||||
|
||||
func (s *Server) adminRoutes() {
|
||||
m := s.mux
|
||||
m.HandleFunc("GET /api/v1/admin/users", s.protected(true, func(w http.ResponseWriter, r *http.Request) {
|
||||
s.list(w, r, `SELECT to_jsonb(u)-'password_hash' FROM users u ORDER BY created_at`)
|
||||
}))
|
||||
m.HandleFunc("POST /api/v1/admin/users", s.protected(true, s.createUser))
|
||||
m.HandleFunc("PATCH /api/v1/admin/users/{id}", s.protected(true, s.updateUser))
|
||||
m.HandleFunc("GET /api/v1/admin/profiles", s.protected(true, func(w http.ResponseWriter, r *http.Request) {
|
||||
s.list(w, r, `SELECT `+profileProjection+` FROM profiles p ORDER BY name`)
|
||||
}))
|
||||
m.HandleFunc("POST /api/v1/admin/profiles", s.protected(true, s.createProfile))
|
||||
m.HandleFunc("PATCH /api/v1/admin/profiles/{id}", s.protected(true, s.updateProfile))
|
||||
m.HandleFunc("POST /api/v1/admin/profiles/{id}/maintenance", s.protected(true, s.maintenance))
|
||||
m.HandleFunc("POST /api/v1/admin/profiles/{id}/publish", s.protected(true, s.publish))
|
||||
m.HandleFunc("POST /api/v1/admin/profiles/{id}/qualify", s.protected(true, s.qualify))
|
||||
m.HandleFunc("GET /api/v1/admin/profiles/{id}/revisions", s.protected(true, func(w http.ResponseWriter, r *http.Request) {
|
||||
s.list(w, r, `SELECT to_jsonb(v)-'source_ref'||jsonb_build_object('os_build',`+revisionOSBuild+`) FROM revisions v WHERE profile_id::text=$1 ORDER BY created_at DESC`, r.PathValue("id"))
|
||||
}))
|
||||
m.HandleFunc("GET /api/v1/admin/profiles/{id}/qualifications", s.protected(true, func(w http.ResponseWriter, r *http.Request) {
|
||||
s.list(w, r, `SELECT to_jsonb(q) FROM qualifications q WHERE profile_id::text=$1 ORDER BY created_at DESC`, r.PathValue("id"))
|
||||
}))
|
||||
m.HandleFunc("GET /api/v1/admin/qualifications/{id}", s.protected(true, func(w http.ResponseWriter, r *http.Request) {
|
||||
v, e := s.queryObject(r.Context(), `SELECT to_jsonb(q) FROM qualifications q WHERE id::text=$1`, r.PathValue("id"))
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 200, v)
|
||||
}))
|
||||
m.HandleFunc("GET /api/v1/admin/bindings", s.protected(true, func(w http.ResponseWriter, r *http.Request) {
|
||||
s.list(w, r, `SELECT to_jsonb(b)||jsonb_build_object('configured',configured AND COALESCE(isolation_expires_at>now(),false),'reason',CASE WHEN configured AND NOT COALESCE(isolation_expires_at>now(),false) THEN 'Isolation proof expired or requires revalidation' ELSE reason END) FROM bindings b ORDER BY owner_id`)
|
||||
}))
|
||||
m.HandleFunc("GET /api/v1/admin/health", s.protected(true, s.health))
|
||||
m.HandleFunc("POST /api/v1/admin/attempts/{id}/release-evidence", s.protected(true, s.releaseEvidence))
|
||||
}
|
||||
func (s *Server) createUser(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
Username string `json:"username"`
|
||||
Password string `json:"password"`
|
||||
Role string `json:"role"`
|
||||
}
|
||||
if !decode(w, r, &in) {
|
||||
return
|
||||
}
|
||||
id, e := CreateUser(r.Context(), s.db, in.Username, in.Password, in.Role)
|
||||
if e != nil {
|
||||
fail(w, 400, "invalid_user", "Username must be unique (3-64 safe characters), password 14-72 bytes, role admin/operator")
|
||||
return
|
||||
}
|
||||
v, e := s.queryObject(r.Context(), `SELECT to_jsonb(u)-'password_hash' FROM users u WHERE id=$1`, id)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 201, v)
|
||||
}
|
||||
func (s *Server) updateUser(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
Role *string `json:"role"`
|
||||
Disabled *bool `json:"disabled"`
|
||||
Password *string `json:"password"`
|
||||
}
|
||||
if !decode(w, r, &in) {
|
||||
return
|
||||
}
|
||||
if in.Role != nil && *in.Role != "admin" && *in.Role != "operator" {
|
||||
fail(w, 400, "invalid_role", "Role must be admin/operator")
|
||||
return
|
||||
}
|
||||
var hash *string
|
||||
if in.Password != nil {
|
||||
if len(*in.Password) < 14 || len(*in.Password) > 72 || strings.TrimSpace(*in.Password) != *in.Password {
|
||||
fail(w, 400, "invalid_password", "Password must be 14-72 bytes without surrounding whitespace")
|
||||
return
|
||||
}
|
||||
b, e := bcrypt.GenerateFromPassword([]byte(*in.Password), 12)
|
||||
if e != nil {
|
||||
fail(w, 500, "internal_error", "Password hashing failed")
|
||||
return
|
||||
}
|
||||
h := string(b)
|
||||
hash = &h
|
||||
}
|
||||
tx, e := s.db.Begin(r.Context())
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(r.Context())
|
||||
if _, e = tx.Exec(r.Context(), `SELECT pg_advisory_xact_lock(74638292)`); e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
var id, role string
|
||||
var disabled bool
|
||||
e = tx.QueryRow(r.Context(), `SELECT id::text,role,disabled FROM users WHERE id::text=$1 FOR UPDATE`, r.PathValue("id")).Scan(&id, &role, &disabled)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if role == "admin" && !disabled && (in.Role != nil && *in.Role != "admin" || in.Disabled != nil && *in.Disabled) {
|
||||
var count int
|
||||
e = tx.QueryRow(r.Context(), `SELECT count(*) FROM users WHERE role='admin' AND NOT disabled`).Scan(&count)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if count <= 1 {
|
||||
fail(w, 409, "last_administrator", "Cannot disable or demote the last administrator")
|
||||
return
|
||||
}
|
||||
}
|
||||
_, e = tx.Exec(r.Context(), `UPDATE users SET role=COALESCE($2,role),disabled=COALESCE($3,disabled),password_hash=COALESCE($4,password_hash) WHERE id=$1`, id, in.Role, in.Disabled, hash)
|
||||
if e == nil {
|
||||
_, e = tx.Exec(r.Context(), `DELETE FROM sessions WHERE user_id=$1`, id)
|
||||
}
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if e = tx.Commit(r.Context()); e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
v, e := s.queryObject(r.Context(), `SELECT to_jsonb(u)-'password_hash' FROM users u WHERE id=$1`, id)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 200, v)
|
||||
}
|
||||
func (s *Server) createProfile(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
Name string `json:"name"`
|
||||
OS string `json:"os"`
|
||||
Architecture string `json:"architecture"`
|
||||
}
|
||||
if !decode(w, r, &in) {
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(in.Name) == "" || len(in.Name) > 120 || len(in.OS) > 120 || in.OS == "" || in.Architecture != "x64" && in.Architecture != "x86" {
|
||||
fail(w, 400, "invalid_profile", "Name, OS and x64/x86 architecture required")
|
||||
return
|
||||
}
|
||||
v, e := s.queryObject(r.Context(), `WITH p AS (INSERT INTO profiles(id,name,os,architecture) VALUES($1,$2,$3,$4) RETURNING *) SELECT `+profileProjection+` FROM p`, store.NewID(), in.Name, in.OS, in.Architecture)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 201, v)
|
||||
}
|
||||
func (s *Server) updateProfile(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
Name *string `json:"name"`
|
||||
Enabled *bool `json:"enabled"`
|
||||
}
|
||||
if !decode(w, r, &in) {
|
||||
return
|
||||
}
|
||||
if in.Name != nil && (strings.TrimSpace(*in.Name) == "" || len(*in.Name) > 120) {
|
||||
fail(w, 400, "invalid_profile", "Invalid name")
|
||||
return
|
||||
}
|
||||
v, e := s.queryObject(r.Context(), `WITH p AS (UPDATE profiles SET name=COALESCE($2,name),enabled=COALESCE($3,enabled) WHERE id::text=$1 RETURNING *) SELECT `+profileProjection+` FROM p`, r.PathValue("id"), in.Name, in.Enabled)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 200, v)
|
||||
}
|
||||
func (s *Server) maintenance(w http.ResponseWriter, r *http.Request) {
|
||||
v, e := s.queryObject(r.Context(), `WITH p AS (UPDATE profiles SET state='maintenance',enabled=false,reason='Maintenance admission closed; drain queued clones before changing source disks' WHERE id::text=$1 RETURNING *) SELECT `+profileProjection+` FROM p`, r.PathValue("id"))
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
var pending int
|
||||
e = s.db.QueryRow(r.Context(), `SELECT count(*) FROM runs r JOIN attempts a ON a.run_id=r.id WHERE r.profile_id::text=$1 AND a.phase IN('queued','provisioning')`, r.PathValue("id")).Scan(&pending)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
v["drain"] = map[string]any{"pending_clones": pending, "safe_to_maintain": pending == 0}
|
||||
jsonResponse(w, 200, v)
|
||||
}
|
||||
func (s *Server) publish(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
RevisionID string `json:"revision_id"`
|
||||
}
|
||||
if !decode(w, r, &in) {
|
||||
return
|
||||
}
|
||||
if !uuidRE.MatchString(in.RevisionID) {
|
||||
fail(w, 400, "invalid_revision", "Revision UUID required")
|
||||
return
|
||||
}
|
||||
tx, e := s.db.Begin(r.Context())
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(r.Context())
|
||||
var profile string
|
||||
e = tx.QueryRow(r.Context(), `SELECT id::text FROM profiles WHERE id::text=$1 FOR UPDATE`, r.PathValue("id")).Scan(&profile)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
var qualified bool
|
||||
e = tx.QueryRow(r.Context(), `SELECT qualification IS NOT NULL AND qualification->>'status'='passed' FROM revisions WHERE id=$1 AND profile_id=$2`, in.RevisionID, profile).Scan(&qualified)
|
||||
if e != nil || !qualified {
|
||||
fail(w, 409, "unqualified_revision", "Worker-validated source and passed qualification required")
|
||||
return
|
||||
}
|
||||
var pending int
|
||||
e = tx.QueryRow(r.Context(), `SELECT count(*) FROM runs r JOIN attempts a ON a.run_id=r.id WHERE r.profile_id=$1 AND r.revision_id<>$2 AND a.phase IN('queued','provisioning')`, profile, in.RevisionID).Scan(&pending)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if pending > 0 {
|
||||
fail(w, 409, "source_not_drained", "Old revision still has unmaterialized attempts")
|
||||
return
|
||||
}
|
||||
_, e = tx.Exec(r.Context(), `UPDATE profiles SET current_revision_id=$2,qualification='qualified',state='published',enabled=true,reason='' WHERE id=$1`, profile, in.RevisionID)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if e = tx.Commit(r.Context()); e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
v, e := s.queryObject(r.Context(), `SELECT `+profileProjection+` FROM profiles p WHERE id=$1`, profile)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 200, v)
|
||||
}
|
||||
func (s *Server) qualify(w http.ResponseWriter, r *http.Request) {
|
||||
tx, e := s.db.Begin(r.Context())
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(r.Context())
|
||||
var id string
|
||||
e = tx.QueryRow(r.Context(), `SELECT id::text FROM profiles WHERE id::text=$1 FOR UPDATE`, r.PathValue("id")).Scan(&id)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
var exists bool
|
||||
e = tx.QueryRow(r.Context(), `SELECT EXISTS(SELECT 1 FROM revisions WHERE profile_id=$1)`, id).Scan(&exists)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if !exists {
|
||||
fail(w, 409, "source_not_configured", "Configure and validate source through worker CLI before qualification")
|
||||
return
|
||||
}
|
||||
e = tx.QueryRow(r.Context(), `SELECT EXISTS(SELECT 1 FROM qualifications WHERE profile_id=$1 AND status IN('queued','running'))`, id).Scan(&exists)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if exists {
|
||||
fail(w, 409, "qualification_active", "A qualification is already active")
|
||||
return
|
||||
}
|
||||
qid := store.NewID()
|
||||
_, e = tx.Exec(r.Context(), `INSERT INTO qualifications(id,profile_id)VALUES($1,$2)`, qid, id)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if e = tx.Commit(r.Context()); e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 202, map[string]any{"id": qid, "status": "queued"})
|
||||
}
|
||||
func (s *Server) health(w http.ResponseWriter, r *http.Request) {
|
||||
var seen any
|
||||
var ready bool
|
||||
var b []byte
|
||||
e := s.db.QueryRow(r.Context(), `SELECT seen_at,ready AND seen_at>now()-interval '45 seconds',blockers FROM heartbeats WHERE name='worker'`).Scan(&seen, &ready, &b)
|
||||
blockers := []string{}
|
||||
if e != nil {
|
||||
blockers = append(blockers, "Execution worker has not connected")
|
||||
} else {
|
||||
_ = json.Unmarshal(b, &blockers)
|
||||
}
|
||||
if !ready && len(blockers) == 0 {
|
||||
blockers = append(blockers, "Execution worker heartbeat is stale")
|
||||
}
|
||||
worker := "unavailable"
|
||||
if ready {
|
||||
worker = "ready"
|
||||
}
|
||||
jsonResponse(w, 200, map[string]any{"database": "ready", "worker": worker, "last_worker_seen": seen, "integration_ready": ready, "blockers": blockers})
|
||||
}
|
||||
func (s *Server) releaseEvidence(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
Confirm bool `json:"confirm"`
|
||||
}
|
||||
if !decode(w, r, &in) {
|
||||
return
|
||||
}
|
||||
if !in.Confirm {
|
||||
fail(w, 400, "confirmation_required", "Explicit confirm true required")
|
||||
return
|
||||
}
|
||||
tag, e := s.db.Exec(r.Context(), `UPDATE attempts SET release_requested=true WHERE id::text=$1 AND phase='finished' AND cleanup IN ('evidence_held','failed') AND EXISTS(SELECT 1 FROM allocations WHERE attempt_id=attempts.id)`, r.PathValue("id"))
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if tag.RowsAffected() != 1 {
|
||||
fail(w, 409, "not_retained", "Attempt is not holding releasable evidence")
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 202, map[string]any{"release_requested": true})
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"otche/internal/store"
|
||||
)
|
||||
|
||||
// This integration test uses its own PostgreSQL schema and never a runtime adapter.
|
||||
func TestAccountLoginLimitAndSessionRevocation(t *testing.T) {
|
||||
url := os.Getenv("OTCHE_TEST_DATABASE_URL")
|
||||
if url == "" {
|
||||
t.Skip("OTCHE_TEST_DATABASE_URL required for isolated PostgreSQL regression")
|
||||
}
|
||||
ctx := context.Background()
|
||||
admin, err := pgxpool.New(ctx, url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Close()
|
||||
schema := "test_" + strings.ReplaceAll(store.NewID(), "-", "")
|
||||
if _, err = admin.Exec(ctx, "CREATE SCHEMA "+schema); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Exec(ctx, "DROP SCHEMA "+schema+" CASCADE")
|
||||
cfg, err := pgxpool.ParseConfig(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg.ConnConfig.RuntimeParams["search_path"] = schema
|
||||
db, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if err = store.Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
password := "Regression-only-long-password"
|
||||
for _, name := range []string{"first-user", "second-user"} {
|
||||
if _, err = CreateUser(ctx, db, name, password, "operator"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
s, err := New(db, Config{Origin: "http://localhost", ArtifactRoot: t.TempDir()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
login := func(name, pass string) *httptest.ResponseRecorder {
|
||||
body, _ := json.Marshal(map[string]string{"username": name, "password": pass})
|
||||
r := httptest.NewRequest("POST", "/api/v1/auth/login", strings.NewReader(string(body)))
|
||||
r.RemoteAddr = "172.20.0.2:80"
|
||||
r.Header.Set("Origin", "http://localhost")
|
||||
w := httptest.NewRecorder()
|
||||
s.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
var cookie *http.Cookie
|
||||
for i := range 17 {
|
||||
w := login("first-user", password)
|
||||
if w.Code != 200 {
|
||||
t.Fatalf("valid login %d locked out: %d %s", i, w.Code, w.Body)
|
||||
}
|
||||
cookie = w.Result().Cookies()[0]
|
||||
}
|
||||
for i := range 15 {
|
||||
w := login("first-user", "wrong password")
|
||||
if w.Code != 401 {
|
||||
t.Fatalf("failed attempt %d: %d", i, w.Code)
|
||||
}
|
||||
}
|
||||
if w := login("first-user", password); w.Code != 429 {
|
||||
t.Fatalf("exhausted account should throttle, got %d", w.Code)
|
||||
}
|
||||
if w := login("second-user", password); w.Code != 200 {
|
||||
t.Fatalf("shared proxy must not lock unrelated account: %d", w.Code)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `UPDATE users SET disabled=true WHERE username='first-user'`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := httptest.NewRequest("GET", "/api/v1/auth/session", nil)
|
||||
r.AddCookie(cookie)
|
||||
w := httptest.NewRecorder()
|
||||
s.ServeHTTP(w, r)
|
||||
if w.Code != 401 {
|
||||
t.Fatalf("disabled user session retained access: %d", w.Code)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `UPDATE users SET role='admin' WHERE username='second-user'`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
elevated := login("second-user", password)
|
||||
adminRequest := httptest.NewRequest("GET", "/api/v1/admin/users", nil)
|
||||
adminRequest.AddCookie(elevated.Result().Cookies()[0])
|
||||
before := httptest.NewRecorder()
|
||||
s.ServeHTTP(before, adminRequest)
|
||||
if before.Code != 200 {
|
||||
t.Fatalf("admin access unavailable: %d", before.Code)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `UPDATE users SET role='operator' WHERE username='second-user'`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
after := httptest.NewRecorder()
|
||||
s.ServeHTTP(after, adminRequest)
|
||||
if after.Code != 403 {
|
||||
t.Fatalf("demoted existing session retained admin: %d", after.Code)
|
||||
}
|
||||
db.Close()
|
||||
w = httptest.NewRecorder()
|
||||
s.ServeHTTP(w, r)
|
||||
if w.Code != 500 {
|
||||
t.Fatalf("database failure disguised as auth failure: %d", w.Code)
|
||||
}
|
||||
fmt.Fprintln(os.Stdout, "17 valid proxied logins; account failure throttling; unrelated user unaffected; revoked session denied; DB failure remains 500")
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"otche/internal/store"
|
||||
)
|
||||
|
||||
func TestBindingExpiryBlocksReadinessAndAdmission(t *testing.T) {
|
||||
url := os.Getenv("OTCHE_TEST_DATABASE_URL")
|
||||
if url == "" {
|
||||
t.Skip("OTCHE_TEST_DATABASE_URL required for isolated PostgreSQL regression")
|
||||
}
|
||||
ctx := context.Background()
|
||||
admin, err := pgxpool.New(ctx, url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Close()
|
||||
schema := "test_" + strings.ReplaceAll(store.NewID(), "-", "")
|
||||
if _, err = admin.Exec(ctx, "CREATE SCHEMA "+schema); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Exec(ctx, "DROP SCHEMA "+schema+" CASCADE")
|
||||
cfg, err := pgxpool.ParseConfig(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg.ConnConfig.RuntimeParams["search_path"] = schema
|
||||
db, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if err = store.Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
owner, err := CreateUser(ctx, db, "expiry-admin", "Regression-only-long-password", "admin")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO bindings(owner_id,pool,iso_storage,disk_storage,node,configured,reason,isolation_expires_at)VALUES($1,'pool','iso','disks','node',true,'',now()+interval '1 hour');`, owner); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO heartbeats(name,seen_at,ready,blockers)VALUES('worker',now(),true,'[]')`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
upload, profile, revision := store.NewID(), store.NewID(), store.NewID()
|
||||
if _, err = db.Exec(ctx, `INSERT INTO uploads(id,owner_id,filename,size,sha256,storage_key)VALUES($1,$2,'benign.exe',1,$3,$4)`, upload, owner, strings.Repeat("a", 64), upload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO profiles(id,name,os,architecture,enabled,qualification,state,current_revision_id)VALUES($1,'isolated expiry fixture','Windows','x64',true,'qualified','published',$2)`, profile, revision); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO revisions(id,profile_id,source_ref,fingerprint,config_digest)VALUES($1,$2,'isolated-test','test','test')`, revision, profile); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s, err := New(db, Config{Origin: "http://localhost", ArtifactRoot: t.TempDir(), MaxQueuedJobs: 20})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
login := httptest.NewRequest("POST", "/api/v1/auth/login", strings.NewReader(`{"username":"expiry-admin","password":"Regression-only-long-password"}`))
|
||||
login.Header.Set("Origin", "http://localhost")
|
||||
logged := httptest.NewRecorder()
|
||||
s.ServeHTTP(logged, login)
|
||||
if logged.Code != 200 {
|
||||
t.Fatalf("login: %d %s", logged.Code, logged.Body)
|
||||
}
|
||||
cookie := logged.Result().Cookies()[0]
|
||||
var session struct {
|
||||
CSRF string `json:"csrf_token"`
|
||||
}
|
||||
if err = json.Unmarshal(logged.Body.Bytes(), &session); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, state := range []struct {
|
||||
expiry string
|
||||
want bool
|
||||
}{{"now()+interval '1 hour'", true}, {"now()-interval '1 second'", false}, {"NULL", false}} {
|
||||
if _, err = db.Exec(ctx, "UPDATE bindings SET configured=true,isolation_expires_at="+state.expiry); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, endpoint := range []string{"/api/v1/dashboard", "/api/v1/admin/bindings"} {
|
||||
r := httptest.NewRequest("GET", endpoint, nil)
|
||||
r.AddCookie(cookie)
|
||||
w := httptest.NewRecorder()
|
||||
s.ServeHTTP(w, r)
|
||||
if w.Code != 200 {
|
||||
t.Fatalf("%s: %d %s", endpoint, w.Code, w.Body)
|
||||
}
|
||||
var body map[string]any
|
||||
if err = json.Unmarshal(w.Body.Bytes(), &body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var got bool
|
||||
if endpoint == "/api/v1/dashboard" {
|
||||
got = body["integration"].(map[string]any)["ready"].(bool)
|
||||
} else {
|
||||
got = body["items"].([]any)[0].(map[string]any)["configured"].(bool)
|
||||
}
|
||||
if got != state.want {
|
||||
t.Fatalf("%s readiness=%v for expiry %s", endpoint, got, state.expiry)
|
||||
}
|
||||
}
|
||||
requestBody, err := json.Marshal(map[string]any{"upload_id": upload, "profile_ids": []string{profile}, "settings": map[string]any{"internet": "offline", "duration_seconds": 30, "filename": "original", "privilege": "user", "args_mode": "none", "args": []string{}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := httptest.NewRequest("POST", "/api/v1/jobs", strings.NewReader(string(requestBody)))
|
||||
r.AddCookie(cookie)
|
||||
r.Header.Set("Origin", "http://localhost")
|
||||
r.Header.Set("X-CSRF-Token", session.CSRF)
|
||||
w := httptest.NewRecorder()
|
||||
s.ServeHTTP(w, r)
|
||||
if state.want {
|
||||
if w.Code != 201 {
|
||||
t.Fatalf("valid isolation proof did not admit owned job: %d %s", w.Code, w.Body)
|
||||
}
|
||||
} else {
|
||||
var rejected struct {
|
||||
Error struct {
|
||||
Code string `json:"code"`
|
||||
} `json:"error"`
|
||||
}
|
||||
if err = json.Unmarshal(w.Body.Bytes(), &rejected); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if w.Code != 409 || rejected.Error.Code != "admission_closed" {
|
||||
t.Fatalf("expired/null proof did not close public admission: %d %s", w.Code, w.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"otche/internal/store"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGrubArchiveOwnerAuthorization(t *testing.T) {
|
||||
url := os.Getenv("OTCHE_TEST_DATABASE_URL")
|
||||
if url == "" {
|
||||
t.Skip("OTCHE_TEST_DATABASE_URL required for isolated PostgreSQL regression")
|
||||
}
|
||||
ctx := context.Background()
|
||||
admin, err := pgxpool.New(ctx, url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Close()
|
||||
schema := "test_" + strings.ReplaceAll(store.NewID(), "-", "")
|
||||
if _, err = admin.Exec(ctx, "CREATE SCHEMA "+schema); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Exec(ctx, "DROP SCHEMA "+schema+" CASCADE")
|
||||
cfg, err := pgxpool.ParseConfig(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg.ConnConfig.RuntimeParams["search_path"] = schema
|
||||
db, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if err = store.Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
owner, err := CreateUser(ctx, db, "grub-owner", "Grub-regression-password-only", "operator")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = CreateUser(ctx, db, "grub-other", "Grub-regression-password-only", "operator"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
upload, job, artifact := store.NewID(), store.NewID(), store.NewID()
|
||||
if _, err = db.Exec(ctx, `INSERT INTO uploads(id,owner_id,filename,size,sha256,storage_key) VALUES($1,$2,'test.bat',1,$3,'upload-' || $4)`, upload, owner, strings.Repeat("a", 64), upload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO jobs(id,owner_id,upload_id,execution_filename,status,settings) VALUES($1,$2,$3,'test.bat','completed','{}')`, job, owner, upload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var archive bytes.Buffer
|
||||
z := zip.NewWriter(&archive)
|
||||
f, err := z.Create("manifest.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.Write([]byte(`{"status":"empty","requested":1,"collected":0}`))
|
||||
if err = z.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := t.TempDir()
|
||||
if err = os.WriteFile(filepath.Join(root, "private.zip"), archive.Bytes(), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sum := sha256.Sum256(archive.Bytes())
|
||||
if _, err = db.Exec(ctx, `INSERT INTO artifacts(id,job_id,kind,filename,content_type,size,sha256,storage_key) VALUES($1,$2,'grub_archive','grub.zip','application/zip',$3,$4,'private.zip')`, artifact, job, archive.Len(), hex.EncodeToString(sum[:])); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s, err := New(db, Config{Origin: "http://localhost", ArtifactRoot: root})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, user := range []string{"grub-owner", "grub-other"} {
|
||||
login := httptest.NewRequest("POST", "/api/v1/auth/login", strings.NewReader(`{"username":"`+user+`","password":"Grub-regression-password-only"}`))
|
||||
login.Header.Set("Origin", "http://localhost")
|
||||
logged := httptest.NewRecorder()
|
||||
s.ServeHTTP(logged, login)
|
||||
if logged.Code != 200 {
|
||||
t.Fatalf("login %s: %d", user, logged.Code)
|
||||
}
|
||||
req := httptest.NewRequest("GET", "/api/v1/artifacts/"+artifact+"/content", nil)
|
||||
req.AddCookie(logged.Result().Cookies()[0])
|
||||
res := httptest.NewRecorder()
|
||||
s.ServeHTTP(res, req)
|
||||
if user == "grub-other" {
|
||||
if res.Code != 404 || bytes.Equal(res.Body.Bytes(), archive.Bytes()) {
|
||||
t.Fatal("foreign owner received private archive")
|
||||
}
|
||||
continue
|
||||
}
|
||||
if res.Code != 200 || !bytes.Equal(res.Body.Bytes(), archive.Bytes()) || !strings.HasPrefix(res.Header().Get("Content-Disposition"), "attachment;") || res.Header().Get("Content-Type") != "application/octet-stream" {
|
||||
t.Fatalf("owner archive not a safe exact-byte attachment: %d %v", res.Code, res.Header())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGrubSettingsManifestBound(t *testing.T) {
|
||||
base := Settings{Internet: "offline", Duration: 30, Filename: "original", Privilege: "user", ArgsMode: "custom", Args: []string{}, GrubPaths: []string{`C:\Logs\100%done{copy}.log`}}
|
||||
if err := base.validate("file.bat"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base.Args = make([]string, 64)
|
||||
for i := range base.Args {
|
||||
base.Args[i] = strings.Repeat("&", 256)
|
||||
}
|
||||
if err := base.validate("file.bat"); err == nil {
|
||||
t.Fatal("escaped serialized manifest overflow was admitted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,214 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"otche/internal/store"
|
||||
)
|
||||
|
||||
func TestJobSummariesIncludeAssignedProfile(t *testing.T) {
|
||||
url := os.Getenv("OTCHE_TEST_DATABASE_URL")
|
||||
if url == "" {
|
||||
t.Skip("OTCHE_TEST_DATABASE_URL required for isolated PostgreSQL regression")
|
||||
}
|
||||
ctx := context.Background()
|
||||
admin, err := pgxpool.New(ctx, url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Close()
|
||||
schema := "test_" + strings.ReplaceAll(store.NewID(), "-", "")
|
||||
if _, err = admin.Exec(ctx, "CREATE SCHEMA "+schema); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Exec(ctx, "DROP SCHEMA "+schema+" CASCADE")
|
||||
cfg, err := pgxpool.ParseConfig(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg.ConnConfig.RuntimeParams["search_path"] = schema
|
||||
db, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if err = store.Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
owner, err := CreateUser(ctx, db, "summary-admin", "Regression-only-long-password", "admin")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
upload, profile, revision, job, run, attempt, artifact := store.NewID(), store.NewID(), store.NewID(), store.NewID(), store.NewID(), store.NewID(), store.NewID()
|
||||
if _, err = db.Exec(ctx, `INSERT INTO uploads(id,owner_id,filename,size,sha256,storage_key) VALUES($1,$2,'panel-bat-smoke.bat',187,$3,$4)`, upload, owner, strings.Repeat("a", 64), upload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO profiles(id,name,os,architecture,enabled,qualification,state,current_revision_id) VALUES($1,'Windows 11 Pro 25H2 - VM7001','Windows 11','x64',true,'qualified','published',$2)`, profile, revision); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO revisions(id,profile_id,source_ref,fingerprint,config_digest) VALUES($1,$2,'win11','fingerprint','config')`, revision, profile); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO jobs(id,owner_id,upload_id,execution_filename,status,settings) VALUES($1,$2,$3,'panel-bat-smoke.bat','completed','{}')`, job, owner, upload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO runs(id,job_id,profile_id,revision_id,status) VALUES($1,$2,$3,$4,'completed')`, run, job, profile, revision); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO attempts(id,run_id,command_id,phase,outcome,findings,telemetry,cleanup,report) VALUES($1,$2,$3,'finished','executed','not_observed','complete','complete','{"execution":{"exit_code":0}}')`, attempt, run, store.NewID()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `INSERT INTO artifacts(id,job_id,attempt_id,kind,filename,content_type,size,sha256,storage_key) VALUES($1,$2,$3,'report','report.json','application/json',27,$4,$5)`, artifact, job, attempt, strings.Repeat("b", 64), artifact); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
s, err := New(db, Config{Origin: "http://localhost", ArtifactRoot: t.TempDir(), MaxQueuedJobs: 20})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
login := httptest.NewRequest("POST", "/api/v1/auth/login", strings.NewReader(`{"username":"summary-admin","password":"Regression-only-long-password"}`))
|
||||
login.Header.Set("Origin", "http://localhost")
|
||||
logged := httptest.NewRecorder()
|
||||
s.ServeHTTP(logged, login)
|
||||
if logged.Code != 200 {
|
||||
t.Fatalf("login: %d %s", logged.Code, logged.Body)
|
||||
}
|
||||
cookie := logged.Result().Cookies()[0]
|
||||
for _, endpoint := range []string{"/api/v1/dashboard", "/api/v1/jobs"} {
|
||||
req := httptest.NewRequest("GET", endpoint, nil)
|
||||
req.AddCookie(cookie)
|
||||
res := httptest.NewRecorder()
|
||||
s.ServeHTTP(res, req)
|
||||
if res.Code != 200 {
|
||||
t.Fatalf("%s: %d %s", endpoint, res.Code, res.Body)
|
||||
}
|
||||
var body struct {
|
||||
RecentJobs []struct {
|
||||
Runs []struct {
|
||||
ProfileName string `json:"profile_name"`
|
||||
} `json:"runs"`
|
||||
} `json:"recent_jobs"`
|
||||
Items []struct {
|
||||
Runs []struct {
|
||||
ProfileName string `json:"profile_name"`
|
||||
} `json:"runs"`
|
||||
} `json:"items"`
|
||||
}
|
||||
if err = json.Unmarshal(res.Body.Bytes(), &body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var runs []struct {
|
||||
ProfileName string `json:"profile_name"`
|
||||
}
|
||||
if endpoint == "/api/v1/dashboard" {
|
||||
if len(body.RecentJobs) != 1 {
|
||||
t.Fatalf("dashboard recent_jobs: %#v", body.RecentJobs)
|
||||
}
|
||||
runs = body.RecentJobs[0].Runs
|
||||
} else {
|
||||
if len(body.Items) != 1 {
|
||||
t.Fatalf("jobs items: %#v", body.Items)
|
||||
}
|
||||
runs = body.Items[0].Runs
|
||||
}
|
||||
if len(runs) != 1 || runs[0].ProfileName != "Windows 11 Pro 25H2 - VM7001" {
|
||||
t.Fatalf("%s profile summary: %#v response=%s", endpoint, runs, res.Body.String())
|
||||
}
|
||||
}
|
||||
detailRequest := httptest.NewRequest("GET", "/api/v1/jobs/"+job, nil)
|
||||
detailRequest.AddCookie(cookie)
|
||||
detailResponse := httptest.NewRecorder()
|
||||
s.ServeHTTP(detailResponse, detailRequest)
|
||||
if detailResponse.Code != 200 {
|
||||
t.Fatalf("job detail: %d %s", detailResponse.Code, detailResponse.Body)
|
||||
}
|
||||
var detail struct {
|
||||
Runs []struct {
|
||||
ProfileName string `json:"profile_name"`
|
||||
Attempts []struct {
|
||||
Report struct {
|
||||
Execution struct {
|
||||
ExitCode *int `json:"exit_code"`
|
||||
} `json:"execution"`
|
||||
} `json:"report"`
|
||||
Artifacts []struct {
|
||||
ID string `json:"id"`
|
||||
} `json:"artifacts"`
|
||||
} `json:"attempts"`
|
||||
} `json:"runs"`
|
||||
}
|
||||
if err = json.Unmarshal(detailResponse.Body.Bytes(), &detail); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(detail.Runs) != 1 || detail.Runs[0].ProfileName != "Windows 11 Pro 25H2 - VM7001" || len(detail.Runs[0].Attempts) != 1 || detail.Runs[0].Attempts[0].Report.Execution.ExitCode == nil || *detail.Runs[0].Attempts[0].Report.Execution.ExitCode != 0 || len(detail.Runs[0].Attempts[0].Artifacts) != 1 || detail.Runs[0].Attempts[0].Artifacts[0].ID != artifact {
|
||||
t.Fatalf("job detail lost evidence: %s", detailResponse.Body.String())
|
||||
}
|
||||
|
||||
// Legacy successful qualification keeps its own recorded control evidence.
|
||||
if _, err = db.Exec(ctx, `UPDATE attempts SET report=report||'{"environment":{"os_build":"26100.1234.26100.1.amd64fre.old"}}'::jsonb WHERE id=$1`, attempt); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `UPDATE revisions SET qualification=jsonb_build_object('status','passed','controls',jsonb_build_array(jsonb_build_object('attempt_id','not-a-uuid'),jsonb_build_object('attempt_id',$2::text))) WHERE id=$1`, revision, attempt); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
newRevision := store.NewID()
|
||||
if _, err = db.Exec(ctx, `INSERT INTO revisions(id,profile_id,source_ref,fingerprint,config_digest,qualification) VALUES($1,$2,'win11','new-fingerprint','new-config','{"status":"passed","environment":{"os_build":"26200.6584.26100.1.amd64fre.new"}}')`, newRevision, profile); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
get := func(endpoint string) map[string]any {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest("GET", endpoint, nil)
|
||||
req.AddCookie(cookie)
|
||||
res := httptest.NewRecorder()
|
||||
s.ServeHTTP(res, req)
|
||||
if res.Code != 200 {
|
||||
t.Fatalf("%s: %d %s", endpoint, res.Code, res.Body)
|
||||
}
|
||||
var body map[string]any
|
||||
if err := json.Unmarshal(res.Body.Bytes(), &body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return body
|
||||
}
|
||||
for _, current := range []string{revision, newRevision} {
|
||||
if _, err = db.Exec(ctx, `UPDATE profiles SET current_revision_id=$2 WHERE id=$1`, profile, current); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
expectedProfileBuild := "26100.1234"
|
||||
if current == newRevision {
|
||||
expectedProfileBuild = "26200.6584"
|
||||
}
|
||||
for _, endpoint := range []string{"/api/v1/profiles", "/api/v1/admin/profiles"} {
|
||||
item := get(endpoint)["items"].([]any)[0].(map[string]any)
|
||||
if item["os_build"] != expectedProfileBuild || item["name"] != "Windows 11 Pro 25H2 - VM7001" {
|
||||
t.Fatalf("profile build/name changed incorrectly: %v", item)
|
||||
}
|
||||
}
|
||||
for _, endpoint := range []string{"/api/v1/jobs/" + job, "/api/v1/jobs", "/api/v1/dashboard"} {
|
||||
body := get(endpoint)
|
||||
if endpoint == "/api/v1/jobs" {
|
||||
body = body["items"].([]any)[0].(map[string]any)
|
||||
}
|
||||
if endpoint == "/api/v1/dashboard" {
|
||||
body = body["recent_jobs"].([]any)[0].(map[string]any)
|
||||
}
|
||||
run := body["runs"].([]any)[0].(map[string]any)
|
||||
if run["revision_id"] != revision || run["os_build"] != "26100.1234" {
|
||||
t.Fatalf("historical run borrowed current profile build: %v", run)
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err = db.Exec(ctx, `UPDATE revisions SET qualification=jsonb_set(qualification,'{status}','"failed"') WHERE id=$1`, newRevision); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if get("/api/v1/profiles")["items"].([]any)[0].(map[string]any)["os_build"] != nil {
|
||||
t.Fatal("failed qualification presented as authoritative Windows build")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,605 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"mime"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"otche/internal/grub"
|
||||
"otche/internal/pve"
|
||||
"otche/internal/store"
|
||||
)
|
||||
|
||||
type Settings struct {
|
||||
Internet string `json:"internet"`
|
||||
Duration int `json:"duration_seconds"`
|
||||
Filename string `json:"filename"`
|
||||
Privilege string `json:"privilege"`
|
||||
ArgsMode string `json:"args_mode"`
|
||||
Args []string `json:"args"`
|
||||
SetZoneID bool `json:"set_zoneid"`
|
||||
DLLMode string `json:"dll_mode"`
|
||||
DLLExport string `json:"dll_export"`
|
||||
Architecture string `json:"architecture"`
|
||||
WSHHost string `json:"wsh_host"`
|
||||
MSIUI string `json:"msi_ui"`
|
||||
GrubPaths []string `json:"grub_paths,omitempty"`
|
||||
}
|
||||
|
||||
var extensions = map[string]bool{".exe": true, ".dll": true, ".scr": true, ".com": true, ".ps1": true, ".vbs": true, ".js": true, ".bat": true, ".cmd": true, ".msi": true}
|
||||
var exportRE = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_@?$]{0,127}$`)
|
||||
|
||||
func validFilename(v string) bool {
|
||||
if v == "" || len(v) > 180 || strings.ContainsAny(v, "\\/:*?\"<>|\x00\r\n") || strings.TrimSpace(v) != v || strings.HasSuffix(v, ".") {
|
||||
return false
|
||||
}
|
||||
for _, c := range v {
|
||||
if c < 32 {
|
||||
return false
|
||||
}
|
||||
}
|
||||
base := strings.ToUpper(strings.SplitN(v, ".", 2)[0])
|
||||
if base == "CON" || base == "PRN" || base == "AUX" || base == "NUL" || regexp.MustCompile(`^(COM|LPT)[1-9]$`).MatchString(base) {
|
||||
return false
|
||||
}
|
||||
return extensions[strings.ToLower(filepath.Ext(v))]
|
||||
}
|
||||
func (v *Settings) validate(filename string) error {
|
||||
if err := grub.Validate(v.GrubPaths); err != nil {
|
||||
return err
|
||||
}
|
||||
if v.Internet != "online" && v.Internet != "offline" {
|
||||
return errors.New("internet must be online/offline")
|
||||
}
|
||||
ok := false
|
||||
for _, d := range []int{30, 60, 90, 120, 180, 300, 600, 900, 1200} {
|
||||
if v.Duration == d {
|
||||
ok = true
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
return errors.New("unsupported duration_seconds")
|
||||
}
|
||||
if v.Filename != "original" && v.Filename != "random" {
|
||||
return errors.New("filename must be original/random")
|
||||
}
|
||||
if v.Privilege != "user" && v.Privilege != "admin" {
|
||||
return errors.New("privilege must be user/admin")
|
||||
}
|
||||
if v.ArgsMode != "none" && v.ArgsMode != "custom" {
|
||||
return errors.New("args_mode must be none/custom")
|
||||
}
|
||||
if v.ArgsMode == "none" && len(v.Args) != 0 {
|
||||
return errors.New("args must be empty with args_mode none")
|
||||
}
|
||||
if len(v.Args) > 64 {
|
||||
return errors.New("too many arguments")
|
||||
}
|
||||
for _, a := range v.Args {
|
||||
if len(a) > 2048 || strings.ContainsAny(a, "\x00\r\n") {
|
||||
return errors.New("invalid argument")
|
||||
}
|
||||
}
|
||||
if v.DLLMode == "" && !strings.EqualFold(filepath.Ext(filename), ".dll") {
|
||||
v.DLLMode = "regsvr32"
|
||||
}
|
||||
if v.Architecture == "" {
|
||||
v.Architecture = "auto"
|
||||
}
|
||||
if v.WSHHost == "" {
|
||||
v.WSHHost = "cscript"
|
||||
}
|
||||
if v.MSIUI == "" {
|
||||
v.MSIUI = "full"
|
||||
}
|
||||
if v.Architecture != "auto" && v.Architecture != "x86" && v.Architecture != "x64" {
|
||||
return errors.New("invalid architecture")
|
||||
}
|
||||
if v.WSHHost != "cscript" && v.WSHHost != "wscript" {
|
||||
return errors.New("invalid WSH host")
|
||||
}
|
||||
if v.MSIUI != "full" && v.MSIUI != "quiet" && v.MSIUI != "passive" {
|
||||
return errors.New("invalid MSI UI")
|
||||
}
|
||||
if strings.EqualFold(filepath.Ext(filename), ".dll") {
|
||||
if v.DLLMode != "regsvr32" && v.DLLMode != "rundll32" {
|
||||
return errors.New("DLL requires explicit compatible handler")
|
||||
}
|
||||
if v.DLLMode == "rundll32" && !exportRE.MatchString(v.DLLExport) {
|
||||
return errors.New("DLL requires explicit compatible export")
|
||||
}
|
||||
}
|
||||
encoded, err := json.Marshal(v)
|
||||
if err != nil || len(encoded) > pve.ControlLimit-2048 {
|
||||
return errors.New("serialized settings and arguments exceed guest manifest transport bound")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (s *Server) jobRoutes() {
|
||||
m := s.mux
|
||||
m.HandleFunc("POST /api/v1/uploads", s.protected(false, s.upload))
|
||||
m.HandleFunc("GET /api/v1/uploads/{id}/content", s.protected(false, s.uploadContent))
|
||||
m.HandleFunc("POST /api/v1/jobs", s.protected(false, s.createJob))
|
||||
m.HandleFunc("GET /api/v1/jobs", s.protected(false, s.jobs))
|
||||
m.HandleFunc("GET /api/v1/jobs/{id}", s.protected(false, s.getJob))
|
||||
m.HandleFunc("POST /api/v1/jobs/{id}/cancel", s.protected(false, s.cancelJob))
|
||||
m.HandleFunc("POST /api/v1/jobs/{id}/retry", s.protected(false, s.retryJob))
|
||||
m.HandleFunc("GET /api/v1/jobs/{id}/events", s.protected(false, s.events))
|
||||
m.HandleFunc("GET /api/v1/jobs/{id}/artifacts", s.protected(false, s.artifacts))
|
||||
m.HandleFunc("GET /api/v1/artifacts/{id}/content", s.protected(false, s.artifactContent))
|
||||
m.HandleFunc("GET /api/v1/attempts/{id}/video", s.protected(false, s.video))
|
||||
m.HandleFunc("GET /api/v1/profiles", s.protected(false, func(w http.ResponseWriter, r *http.Request) {
|
||||
s.list(w, r, `SELECT `+profileProjection+` FROM profiles p ORDER BY name`)
|
||||
}))
|
||||
m.HandleFunc("GET /api/v1/dashboard", s.protected(false, s.dashboard))
|
||||
}
|
||||
func (s *Server) upload(w http.ResponseWriter, r *http.Request) {
|
||||
a := user(r)
|
||||
name, e := url.PathUnescape(r.Header.Get("X-Filename"))
|
||||
if e != nil || !validFilename(name) {
|
||||
fail(w, 400, "invalid_filename", "A safe filename with one of the ten supported extensions is required")
|
||||
return
|
||||
}
|
||||
tx, e := s.db.Begin(r.Context())
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(r.Context())
|
||||
var id string
|
||||
if e = tx.QueryRow(r.Context(), `SELECT id::text FROM users WHERE id=$1 FOR UPDATE`, a.ID).Scan(&id); e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
var used int64
|
||||
e = tx.QueryRow(r.Context(), `SELECT COALESCE((SELECT sum(size) FROM uploads WHERE owner_id=$1),0)+COALESCE((SELECT sum(a.size) FROM artifacts a JOIN jobs j ON j.id=a.job_id WHERE j.owner_id=$1),0)`, a.ID).Scan(&used)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
remaining := s.cfg.MaxOwnerBytes - used
|
||||
if remaining <= 0 {
|
||||
fail(w, 429, "quota_exceeded", "Owner storage quota reached")
|
||||
return
|
||||
}
|
||||
limit := s.cfg.MaxUploadBytes
|
||||
if remaining < limit {
|
||||
limit = remaining
|
||||
}
|
||||
if r.ContentLength > limit {
|
||||
fail(w, 413, "upload_too_large", "Upload exceeds quota")
|
||||
return
|
||||
}
|
||||
uploadID := store.NewID()
|
||||
key := "uploads/" + a.ID + "/" + uploadID
|
||||
path := filepath.Join(s.cfg.ArtifactRoot, filepath.FromSlash(key))
|
||||
if e = os.MkdirAll(filepath.Dir(path), 0700); e != nil {
|
||||
fail(w, 500, "storage_error", "Private storage unavailable")
|
||||
return
|
||||
}
|
||||
f, e := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
|
||||
if e != nil {
|
||||
fail(w, 500, "storage_error", "Private storage unavailable")
|
||||
return
|
||||
}
|
||||
keep := false
|
||||
defer func() {
|
||||
f.Close()
|
||||
if !keep {
|
||||
os.Remove(path)
|
||||
}
|
||||
}()
|
||||
h := sha256.New()
|
||||
n, e := io.Copy(io.MultiWriter(f, h), http.MaxBytesReader(w, r.Body, limit))
|
||||
if e != nil {
|
||||
fail(w, 413, "upload_failed", "Upload incomplete or exceeds limit")
|
||||
return
|
||||
}
|
||||
if n == 0 {
|
||||
fail(w, 400, "empty_upload", "Empty uploads are not accepted")
|
||||
return
|
||||
}
|
||||
if e = f.Sync(); e != nil {
|
||||
fail(w, 500, "storage_error", "Could not commit upload")
|
||||
return
|
||||
}
|
||||
if e = f.Close(); e != nil {
|
||||
fail(w, 500, "storage_error", "Could not finalize upload")
|
||||
return
|
||||
}
|
||||
sum := hex.EncodeToString(h.Sum(nil))
|
||||
var created any
|
||||
e = tx.QueryRow(r.Context(), `INSERT INTO uploads(id,owner_id,filename,size,sha256,storage_key) VALUES($1,$2,$3,$4,$5,$6) RETURNING created_at`, uploadID, a.ID, name, n, sum, key).Scan(&created)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if e = tx.Commit(r.Context()); e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
keep = true
|
||||
jsonResponse(w, 201, map[string]any{"id": uploadID, "filename": name, "size": n, "sha256": sum, "created_at": created})
|
||||
}
|
||||
func (s *Server) servePrivate(w http.ResponseWriter, r *http.Request, key, name, ct string) {
|
||||
root, e := os.OpenRoot(s.cfg.ArtifactRoot)
|
||||
if e != nil {
|
||||
fail(w, 500, "storage_error", "Storage unavailable")
|
||||
return
|
||||
}
|
||||
defer root.Close()
|
||||
f, e := root.Open(filepath.FromSlash(key))
|
||||
if e != nil {
|
||||
fail(w, 404, "artifact_unavailable", "Artifact unavailable")
|
||||
return
|
||||
}
|
||||
defer f.Close()
|
||||
st, e := f.Stat()
|
||||
if e != nil || !st.Mode().IsRegular() {
|
||||
fail(w, 404, "artifact_unavailable", "Artifact unavailable")
|
||||
return
|
||||
}
|
||||
disposition := "attachment"
|
||||
if ct == "video/mp4" {
|
||||
disposition = "inline"
|
||||
}
|
||||
w.Header().Set("Content-Disposition", mime.FormatMediaType(disposition, map[string]string{"filename": name}))
|
||||
if ct != "video/mp4" {
|
||||
ct = "application/octet-stream"
|
||||
}
|
||||
w.Header().Set("Content-Type", ct)
|
||||
http.ServeContent(w, r, name, st.ModTime(), f)
|
||||
}
|
||||
func (s *Server) uploadContent(w http.ResponseWriter, r *http.Request) {
|
||||
a := user(r)
|
||||
var key, name string
|
||||
e := s.db.QueryRow(r.Context(), `SELECT storage_key,filename FROM uploads WHERE id::text=$1 AND(owner_id=$2 OR $3='admin')`, r.PathValue("id"), a.ID, a.Role).Scan(&key, &name)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
s.servePrivate(w, r, key, name, "application/octet-stream")
|
||||
}
|
||||
func (s *Server) artifactContent(w http.ResponseWriter, r *http.Request) {
|
||||
a := user(r)
|
||||
var key, name, ct string
|
||||
e := s.db.QueryRow(r.Context(), `SELECT a.storage_key,a.filename,a.content_type FROM artifacts a JOIN jobs j ON j.id=a.job_id WHERE a.id::text=$1 AND(j.owner_id=$2 OR $3='admin')`, r.PathValue("id"), a.ID, a.Role).Scan(&key, &name, &ct)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
s.servePrivate(w, r, key, name, ct)
|
||||
}
|
||||
func (s *Server) createJob(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
UploadID string `json:"upload_id"`
|
||||
ProfileIDs []string `json:"profile_ids"`
|
||||
Settings Settings `json:"settings"`
|
||||
}
|
||||
if !decode(w, r, &in) {
|
||||
return
|
||||
}
|
||||
if !uuidRE.MatchString(in.UploadID) || len(in.ProfileIDs) == 0 || len(in.ProfileIDs) > 12 {
|
||||
fail(w, 400, "invalid_request", "Select one to twelve profiles and an upload")
|
||||
return
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, p := range in.ProfileIDs {
|
||||
if !uuidRE.MatchString(p) || seen[p] {
|
||||
fail(w, 400, "invalid_request", "Profile identifiers must be unique UUIDs")
|
||||
return
|
||||
}
|
||||
seen[p] = true
|
||||
}
|
||||
a := user(r)
|
||||
tx, e := s.db.Begin(r.Context())
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(r.Context())
|
||||
var name string
|
||||
e = tx.QueryRow(r.Context(), `SELECT filename FROM uploads WHERE id=$1 AND owner_id=$2`, in.UploadID, a.ID).Scan(&name)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if e = in.Settings.validate(name); e != nil {
|
||||
fail(w, 400, "invalid_settings", e.Error())
|
||||
return
|
||||
}
|
||||
if e = s.admit(r, tx, a.ID); e != nil {
|
||||
fail(w, 409, "admission_closed", e.Error())
|
||||
return
|
||||
}
|
||||
revisions := map[string]string{}
|
||||
for _, id := range in.ProfileIDs {
|
||||
var revision string
|
||||
var valid bool
|
||||
e = tx.QueryRow(r.Context(), `SELECT COALESCE(current_revision_id::text,''),enabled AND qualification='qualified' AND state='published' AND ($2='offline' OR online_available) FROM profiles WHERE id=$1 FOR SHARE`, id, in.Settings.Internet).Scan(&revision, &valid)
|
||||
if e != nil || !valid || revision == "" {
|
||||
fail(w, 409, "profile_unavailable", "Selected profile is unqualified, in maintenance, or does not support the requested network")
|
||||
return
|
||||
}
|
||||
revisions[id] = revision
|
||||
}
|
||||
execName := name
|
||||
if in.Settings.Filename == "random" {
|
||||
execName = store.NewID() + filepath.Ext(name)
|
||||
}
|
||||
id := store.NewID()
|
||||
settings, _ := json.Marshal(in.Settings)
|
||||
_, e = tx.Exec(r.Context(), `INSERT INTO jobs(id,owner_id,upload_id,execution_filename,settings)VALUES($1,$2,$3,$4,$5)`, id, a.ID, in.UploadID, execName, settings)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
for _, p := range in.ProfileIDs {
|
||||
run := store.NewID()
|
||||
if _, e = tx.Exec(r.Context(), `INSERT INTO runs(id,job_id,profile_id,revision_id)VALUES($1,$2,$3,$4)`, run, id, p, revisions[p]); e == nil {
|
||||
_, e = tx.Exec(r.Context(), `INSERT INTO attempts(id,run_id,command_id)VALUES($1,$2,$3)`, store.NewID(), run, store.NewID())
|
||||
}
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
}
|
||||
_, e = tx.Exec(r.Context(), `INSERT INTO events(job_id,kind,message)VALUES($1,'queued','Job admitted with immutable settings and source revisions')`, id)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if e = tx.Commit(r.Context()); e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
s.respondJob(w, r, id, 201)
|
||||
}
|
||||
func (s *Server) admit(r *http.Request, tx pgx.Tx, owner string) error {
|
||||
var id string
|
||||
if e := tx.QueryRow(r.Context(), `SELECT id::text FROM users WHERE id=$1 FOR UPDATE`, owner).Scan(&id); e != nil {
|
||||
return e
|
||||
}
|
||||
var configured bool
|
||||
e := tx.QueryRow(r.Context(), `SELECT configured AND COALESCE(isolation_expires_at>now(),false) FROM bindings WHERE owner_id=$1`, owner).Scan(&configured)
|
||||
if e != nil || !configured {
|
||||
return errors.New("Owner PVE isolation and credentials are not configured")
|
||||
}
|
||||
var workerReady bool
|
||||
e = tx.QueryRow(r.Context(), `SELECT COALESCE(bool_or(ready AND seen_at>now()-interval '45 seconds'),false) FROM heartbeats WHERE name='worker'`).Scan(&workerReady)
|
||||
if e != nil || !workerReady {
|
||||
return errors.New("Execution worker is not ready")
|
||||
}
|
||||
var count int
|
||||
e = tx.QueryRow(r.Context(), `SELECT count(*) FROM jobs WHERE owner_id=$1 AND status IN('queued','running')`, owner).Scan(&count)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if count >= s.cfg.MaxQueuedJobs {
|
||||
return errors.New("Owner queue quota reached")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
const jobRunsSummary = `COALESCE((SELECT jsonb_agg(` + runProjection + ` ORDER BY r.id) FROM runs r JOIN profiles p ON p.id=r.profile_id WHERE r.job_id=j.id),'[]'::jsonb)`
|
||||
|
||||
const jobQuery = `SELECT to_jsonb(j)||jsonb_build_object('filename',u.filename,'sha256',u.sha256,'size',u.size,'runs',COALESCE((SELECT jsonb_agg(` + runProjection + `||jsonb_build_object('attempts',COALESCE((SELECT jsonb_agg((to_jsonb(a)-'lease_owner'-'lease_until'-'release_requested')||jsonb_build_object('allocation',CASE WHEN $3='admin' THEN(SELECT jsonb_build_object('id',v.id,'node',v.node,'vmid',v.vmid,'state',v.state) FROM allocations v WHERE v.attempt_id=a.id) ELSE NULL END,'artifacts',COALESCE((SELECT jsonb_agg(to_jsonb(ar)-'storage_key'||jsonb_build_object('url','/api/v1/artifacts/'||ar.id||'/content')) FROM artifacts ar WHERE ar.attempt_id=a.id),'[]'::jsonb)) ORDER BY a.created_at) FROM attempts a WHERE a.run_id=r.id),'[]'::jsonb))) FROM runs r JOIN profiles p ON p.id=r.profile_id WHERE r.job_id=j.id),'[]'::jsonb)) FROM jobs j JOIN uploads u ON u.id=j.upload_id WHERE j.id::text=$1 AND(j.owner_id=$2 OR $3='admin')`
|
||||
|
||||
func (s *Server) respondJob(w http.ResponseWriter, r *http.Request, id string, status int) {
|
||||
a := user(r)
|
||||
v, e := s.queryObject(r.Context(), jobQuery, id, a.ID, a.Role)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, status, v)
|
||||
}
|
||||
func (s *Server) getJob(w http.ResponseWriter, r *http.Request) {
|
||||
s.respondJob(w, r, r.PathValue("id"), 200)
|
||||
}
|
||||
func (s *Server) ownJob(r *http.Request) (string, error) {
|
||||
a := user(r)
|
||||
var id string
|
||||
e := s.db.QueryRow(r.Context(), `SELECT id::text FROM jobs WHERE id::text=$1 AND(owner_id=$2 OR $3='admin')`, r.PathValue("id"), a.ID, a.Role).Scan(&id)
|
||||
return id, e
|
||||
}
|
||||
func (s *Server) jobs(w http.ResponseWriter, r *http.Request) {
|
||||
a := user(r)
|
||||
page, _ := strconv.Atoi(r.URL.Query().Get("page"))
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
size, _ := strconv.Atoi(r.URL.Query().Get("page_size"))
|
||||
if size < 1 {
|
||||
size = 25
|
||||
}
|
||||
if size > 100 {
|
||||
size = 100
|
||||
}
|
||||
if page > 100000 {
|
||||
fail(w, 400, "invalid_page", "Page too large")
|
||||
return
|
||||
}
|
||||
q := r.URL.Query().Get("q")
|
||||
if len(q) > 128 {
|
||||
fail(w, 400, "invalid_query", "Search too long")
|
||||
return
|
||||
}
|
||||
status := r.URL.Query().Get("status")
|
||||
var total int
|
||||
where := ` FROM jobs j JOIN uploads u ON u.id=j.upload_id WHERE(j.owner_id=$1 OR $2='admin') AND($3='' OR u.filename ILIKE '%'||$3||'%' OR u.sha256=$3) AND($4='' OR j.status=$4)`
|
||||
e := s.db.QueryRow(r.Context(), `SELECT count(*)`+where, a.ID, a.Role, q, status).Scan(&total)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
v, e := s.queryList(r.Context(), `SELECT to_jsonb(j)||jsonb_build_object('filename',u.filename,'sha256',u.sha256,'size',u.size,'runs',`+jobRunsSummary+`)`+where+` ORDER BY j.created_at DESC LIMIT $5 OFFSET $6`, a.ID, a.Role, q, status, size, (page-1)*size)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
var next any
|
||||
if page*size < total {
|
||||
next = page + 1
|
||||
}
|
||||
jsonResponse(w, 200, map[string]any{"items": v, "total": total, "page": page, "page_size": size, "next_page": next})
|
||||
}
|
||||
func (s *Server) cancelJob(w http.ResponseWriter, r *http.Request) {
|
||||
id, e := s.ownJob(r)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
_, e = s.db.Exec(r.Context(), `UPDATE jobs SET cancel_requested=true,updated_at=now() WHERE id=$1 AND status IN('queued','running')`, id)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
s.respondJob(w, r, id, 200)
|
||||
}
|
||||
func (s *Server) retryJob(w http.ResponseWriter, r *http.Request) {
|
||||
id, e := s.ownJob(r)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
tx, e := s.db.Begin(r.Context())
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(r.Context())
|
||||
var status, owner string
|
||||
e = tx.QueryRow(r.Context(), `SELECT status,owner_id::text FROM jobs WHERE id=$1 FOR UPDATE`, id).Scan(&status, &owner)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if status == "queued" || status == "running" {
|
||||
fail(w, 409, "job_active", "An active job cannot be retried")
|
||||
return
|
||||
}
|
||||
if e = s.admit(r, tx, owner); e != nil {
|
||||
fail(w, 409, "admission_closed", e.Error())
|
||||
return
|
||||
}
|
||||
rows, e := tx.Query(r.Context(), `SELECT r.id::text,p.enabled AND p.state='published' AND p.qualification='qualified' AND p.current_revision_id=r.revision_id FROM runs r JOIN profiles p ON p.id=r.profile_id WHERE r.job_id=$1 FOR SHARE OF p`, id)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
ids := []string{}
|
||||
valid := true
|
||||
for rows.Next() {
|
||||
var run string
|
||||
var ok bool
|
||||
if e = rows.Scan(&run, &ok); e != nil {
|
||||
break
|
||||
}
|
||||
valid = valid && ok
|
||||
ids = append(ids, run)
|
||||
}
|
||||
rows.Close()
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if !valid {
|
||||
fail(w, 409, "revision_unavailable", "Original source revision is no longer admitted; retry cannot silently change revision")
|
||||
return
|
||||
}
|
||||
for _, run := range ids {
|
||||
if _, e = tx.Exec(r.Context(), `INSERT INTO attempts(id,run_id,command_id)VALUES($1,$2,$3)`, store.NewID(), run, store.NewID()); e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
}
|
||||
_, e = tx.Exec(r.Context(), `UPDATE runs SET status='queued' WHERE job_id=$1`, id)
|
||||
if e == nil {
|
||||
_, e = tx.Exec(r.Context(), `UPDATE jobs SET status='queued',cancel_requested=false,updated_at=now() WHERE id=$1`, id)
|
||||
}
|
||||
if e == nil {
|
||||
_, e = tx.Exec(r.Context(), `INSERT INTO events(job_id,kind,message)VALUES($1,'retry','New attempts queued; prior attempts preserved')`, id)
|
||||
}
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
if e = tx.Commit(r.Context()); e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
s.respondJob(w, r, id, 200)
|
||||
}
|
||||
func (s *Server) events(w http.ResponseWriter, r *http.Request) {
|
||||
id, e := s.ownJob(r)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
after, _ := strconv.ParseInt(r.URL.Query().Get("after"), 10, 64)
|
||||
s.list(w, r, `SELECT to_jsonb(e) FROM events e WHERE job_id=$1 AND id>$2 ORDER BY id LIMIT 500`, id, after)
|
||||
}
|
||||
func (s *Server) artifacts(w http.ResponseWriter, r *http.Request) {
|
||||
id, e := s.ownJob(r)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
s.list(w, r, `SELECT to_jsonb(a)-'storage_key'||jsonb_build_object('url','/api/v1/artifacts/'||a.id||'/content') FROM artifacts a WHERE job_id=$1 ORDER BY created_at`, id)
|
||||
}
|
||||
func (s *Server) video(w http.ResponseWriter, r *http.Request) {
|
||||
a := user(r)
|
||||
v, e := s.queryObject(r.Context(), `SELECT a.video FROM attempts a JOIN runs r ON r.id=a.run_id JOIN jobs j ON j.id=r.job_id WHERE a.id::text=$1 AND(j.owner_id=$2 OR $3='admin')`, r.PathValue("id"), a.ID, a.Role)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 200, v)
|
||||
}
|
||||
func (s *Server) dashboard(w http.ResponseWriter, r *http.Request) {
|
||||
a := user(r)
|
||||
metrics, e := s.queryObject(r.Context(), `SELECT jsonb_build_object('jobs',count(*),'queued',count(*)FILTER(WHERE status='queued'),'running',count(*)FILTER(WHERE status='running'),'completed',count(*)FILTER(WHERE status='completed'),'failed',count(*)FILTER(WHERE status='failed'),'cancelled',count(*)FILTER(WHERE status='cancelled'),'detected',count(*)FILTER(WHERE EXISTS(SELECT 1 FROM runs r JOIN attempts a ON a.run_id=r.id WHERE r.job_id=j.id AND a.findings='detected'))) FROM jobs j WHERE owner_id=$1 OR $2='admin'`, a.ID, a.Role)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
recent, e := s.queryList(r.Context(), `SELECT to_jsonb(j)||jsonb_build_object('filename',u.filename,'sha256',u.sha256,'size',u.size,'runs',`+jobRunsSummary+`) FROM jobs j JOIN uploads u ON u.id=j.upload_id WHERE j.owner_id=$1 OR $2='admin' ORDER BY j.created_at DESC LIMIT 5`, a.ID, a.Role)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
health := s.integration(r)
|
||||
jsonResponse(w, 200, map[string]any{"metrics": metrics, "recent_jobs": recent, "queue": map[string]any{"queued": metrics["queued"], "running": metrics["running"]}, "integration": health})
|
||||
}
|
||||
func (s *Server) integration(r *http.Request) map[string]any {
|
||||
var ready bool
|
||||
var b []byte
|
||||
e := s.db.QueryRow(r.Context(), `SELECT ready AND seen_at>now()-interval '45 seconds',blockers FROM heartbeats WHERE name='worker'`).Scan(&ready, &b)
|
||||
blockers := []string{}
|
||||
if e != nil {
|
||||
blockers = append(blockers, "Execution worker has not connected")
|
||||
} else {
|
||||
_ = json.Unmarshal(b, &blockers)
|
||||
if !ready && len(blockers) == 0 {
|
||||
blockers = append(blockers, "Execution worker is unavailable")
|
||||
}
|
||||
}
|
||||
var configured bool
|
||||
if e = s.db.QueryRow(r.Context(), `SELECT configured AND COALESCE(isolation_expires_at>now(),false) FROM bindings WHERE owner_id=$1`, user(r).ID).Scan(&configured); e != nil || !configured {
|
||||
ready = false
|
||||
blockers = append(blockers, "Owner isolation binding is missing, invalid or expired")
|
||||
}
|
||||
return map[string]any{"ready": ready, "blockers": blockers}
|
||||
}
|
||||
@@ -0,0 +1,319 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"golang.org/x/crypto/bcrypt"
|
||||
"otche/internal/store"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Origin, ArtifactRoot string
|
||||
SecureCookies bool
|
||||
MaxUploadBytes, MaxOwnerBytes int64
|
||||
MaxQueuedJobs int
|
||||
}
|
||||
type Server struct {
|
||||
db *pgxpool.Pool
|
||||
cfg Config
|
||||
mux *http.ServeMux
|
||||
dummyHash []byte
|
||||
}
|
||||
type identity struct{ ID, Username, Role, CSRF, TokenHash string }
|
||||
type authKey struct{}
|
||||
|
||||
var uuidRE = regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)
|
||||
var usernameRE = regexp.MustCompile(`^[A-Za-z0-9_.-]{3,64}$`)
|
||||
|
||||
func New(db *pgxpool.Pool, c Config) (*Server, error) {
|
||||
if c.Origin == "" || c.ArtifactRoot == "" {
|
||||
return nil, errors.New("origin and artifact root required")
|
||||
}
|
||||
if c.MaxUploadBytes <= 0 {
|
||||
c.MaxUploadBytes = 512 << 20
|
||||
}
|
||||
if c.MaxOwnerBytes <= 0 {
|
||||
c.MaxOwnerBytes = 8 << 30
|
||||
}
|
||||
if c.MaxQueuedJobs <= 0 {
|
||||
c.MaxQueuedJobs = 20
|
||||
}
|
||||
if e := os.MkdirAll(c.ArtifactRoot, 0700); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
h, e := bcrypt.GenerateFromPassword([]byte(randomToken()), 12)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
s := &Server{db: db, cfg: c, mux: http.NewServeMux(), dummyHash: h}
|
||||
s.routes()
|
||||
return s, nil
|
||||
}
|
||||
func randomToken() string {
|
||||
b := make([]byte, 32)
|
||||
if _, e := rand.Read(b); e != nil {
|
||||
panic(e)
|
||||
}
|
||||
return hex.EncodeToString(b)
|
||||
}
|
||||
func hashToken(v string) string { h := sha256.Sum256([]byte(v)); return hex.EncodeToString(h[:]) }
|
||||
func jsonResponse(w http.ResponseWriter, status int, v any) {
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
func fail(w http.ResponseWriter, status int, code, message string) {
|
||||
jsonResponse(w, status, map[string]any{"error": map[string]string{"code": code, "message": message}})
|
||||
}
|
||||
func decode(w http.ResponseWriter, r *http.Request, v any) bool {
|
||||
r.Body = http.MaxBytesReader(w, r.Body, 64<<10)
|
||||
d := json.NewDecoder(r.Body)
|
||||
d.DisallowUnknownFields()
|
||||
if e := d.Decode(v); e != nil {
|
||||
fail(w, 400, "invalid_request", "Invalid JSON request")
|
||||
return false
|
||||
}
|
||||
if e := d.Decode(new(any)); e != io.EOF {
|
||||
fail(w, 400, "invalid_request", "Expected one JSON value")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
func user(r *http.Request) identity { return r.Context().Value(authKey{}).(identity) }
|
||||
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
w.Header().Set("Referrer-Policy", "no-referrer")
|
||||
w.Header().Set("Content-Security-Policy", "default-src 'none'; sandbox")
|
||||
defer func() {
|
||||
if recover() != nil {
|
||||
slog.Error("API request panicked")
|
||||
fail(w, 500, "internal_error", "Request failed")
|
||||
}
|
||||
}()
|
||||
s.mux.ServeHTTP(w, r)
|
||||
}
|
||||
func (s *Server) origin(w http.ResponseWriter, r *http.Request) bool {
|
||||
if r.Header.Get("Origin") != s.cfg.Origin {
|
||||
fail(w, 403, "origin_rejected", "Origin is not allowed")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
func (s *Server) protected(admin bool, h http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
c, err := r.Cookie("otche_session")
|
||||
if err != nil || len(c.Value) != 64 {
|
||||
fail(w, 401, "unauthenticated", "Sign in required")
|
||||
return
|
||||
}
|
||||
a := identity{TokenHash: hashToken(c.Value)}
|
||||
err = s.db.QueryRow(r.Context(), `SELECT u.id::text,u.username,u.role,s.csrf_token FROM sessions s JOIN users u ON u.id=s.user_id WHERE s.token_hash=$1 AND s.expires_at>now() AND NOT u.disabled`, a.TokenHash).Scan(&a.ID, &a.Username, &a.Role, &a.CSRF)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
fail(w, 401, "unauthenticated", "Session expired")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
s.dbError(w, err)
|
||||
return
|
||||
}
|
||||
if admin && a.Role != "admin" {
|
||||
fail(w, 403, "forbidden", "Administrator required")
|
||||
return
|
||||
}
|
||||
if r.Method != "GET" && r.Method != "HEAD" {
|
||||
if !s.origin(w, r) {
|
||||
return
|
||||
}
|
||||
if subtle.ConstantTimeCompare([]byte(r.Header.Get("X-CSRF-Token")), []byte(a.CSRF)) != 1 {
|
||||
fail(w, 403, "csrf_rejected", "CSRF token required")
|
||||
return
|
||||
}
|
||||
}
|
||||
h(w, r.WithContext(context.WithValue(r.Context(), authKey{}, a)))
|
||||
}
|
||||
}
|
||||
func (s *Server) dbError(w http.ResponseWriter, e error) {
|
||||
if errors.Is(e, pgx.ErrNoRows) {
|
||||
fail(w, 404, "not_found", "Resource not found")
|
||||
return
|
||||
}
|
||||
slog.Error("database operation failed", "type", "query")
|
||||
fail(w, 500, "internal_error", "Database operation failed")
|
||||
}
|
||||
func (s *Server) queryObject(ctx context.Context, q string, args ...any) (map[string]any, error) {
|
||||
var b []byte
|
||||
e := s.db.QueryRow(ctx, q, args...).Scan(&b)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
var v map[string]any
|
||||
e = json.Unmarshal(b, &v)
|
||||
return v, e
|
||||
}
|
||||
func (s *Server) queryList(ctx context.Context, q string, args ...any) ([]map[string]any, error) {
|
||||
rows, e := s.db.Query(ctx, q, args...)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
defer rows.Close()
|
||||
out := []map[string]any{}
|
||||
for rows.Next() {
|
||||
var b []byte
|
||||
if e = rows.Scan(&b); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
var v map[string]any
|
||||
if e = json.Unmarshal(b, &v); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
out = append(out, v)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
func (s *Server) list(w http.ResponseWriter, r *http.Request, q string, args ...any) {
|
||||
v, e := s.queryList(r.Context(), q, args...)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 200, map[string]any{"items": v})
|
||||
}
|
||||
func (s *Server) routes() {
|
||||
m := s.mux
|
||||
m.HandleFunc("GET /api/v1/health", func(w http.ResponseWriter, r *http.Request) {
|
||||
ctx, c := context.WithTimeout(r.Context(), 2*time.Second)
|
||||
defer c()
|
||||
if s.db.Ping(ctx) != nil {
|
||||
fail(w, 503, "unavailable", "Database unavailable")
|
||||
return
|
||||
}
|
||||
jsonResponse(w, 200, map[string]string{"status": "ok"})
|
||||
})
|
||||
m.HandleFunc("POST /api/v1/auth/login", s.login)
|
||||
m.HandleFunc("GET /api/v1/auth/session", s.protected(false, s.session))
|
||||
m.HandleFunc("POST /api/v1/auth/logout", s.protected(false, s.logout))
|
||||
s.jobRoutes()
|
||||
s.adminRoutes()
|
||||
}
|
||||
func (s *Server) session(w http.ResponseWriter, r *http.Request) {
|
||||
a := user(r)
|
||||
jsonResponse(w, 200, map[string]any{"user": map[string]string{"id": a.ID, "username": a.Username, "role": a.Role}, "csrf_token": a.CSRF})
|
||||
}
|
||||
func (s *Server) login(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.origin(w, r) {
|
||||
return
|
||||
}
|
||||
var in struct {
|
||||
Username string `json:"username"`
|
||||
Password string `json:"password"`
|
||||
}
|
||||
if !decode(w, r, &in) {
|
||||
return
|
||||
}
|
||||
if !usernameRE.MatchString(in.Username) || len(in.Password) > 72 {
|
||||
fail(w, 401, "invalid_credentials", "Invalid username or password")
|
||||
return
|
||||
}
|
||||
// Account-scoped failures cannot collapse every proxied user into nginx's IP.
|
||||
// No client-supplied forwarding header is trusted for authentication policy.
|
||||
tx, err := s.db.Begin(r.Context())
|
||||
if err != nil {
|
||||
s.dbError(w, err)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(r.Context())
|
||||
key := hashToken("account:" + strings.ToLower(in.Username))
|
||||
_, err = tx.Exec(r.Context(), `INSERT INTO login_limits(key,failures,reset_at) VALUES($1,0,now()+interval '15 minutes') ON CONFLICT DO NOTHING`, key)
|
||||
if err != nil {
|
||||
s.dbError(w, err)
|
||||
return
|
||||
}
|
||||
var failures int
|
||||
err = tx.QueryRow(r.Context(), `SELECT CASE WHEN reset_at>now() THEN failures ELSE 0 END FROM login_limits WHERE key=$1 FOR UPDATE`, key).Scan(&failures)
|
||||
if err != nil {
|
||||
s.dbError(w, err)
|
||||
return
|
||||
}
|
||||
if failures >= 15 {
|
||||
w.Header().Set("Retry-After", "900")
|
||||
fail(w, 429, "rate_limited", "Too many failed login attempts for this account")
|
||||
return
|
||||
}
|
||||
a := identity{}
|
||||
var hash string
|
||||
var disabled bool
|
||||
err = tx.QueryRow(r.Context(), `SELECT id::text,username,role,password_hash,disabled FROM users WHERE username=$1`, in.Username).Scan(&a.ID, &a.Username, &a.Role, &hash, &disabled)
|
||||
missing := errors.Is(err, pgx.ErrNoRows)
|
||||
if err != nil && !missing {
|
||||
s.dbError(w, err)
|
||||
return
|
||||
}
|
||||
if missing {
|
||||
hash = string(s.dummyHash)
|
||||
}
|
||||
match := bcrypt.CompareHashAndPassword([]byte(hash), []byte(in.Password)) == nil
|
||||
if missing || !match || disabled {
|
||||
_, err = tx.Exec(r.Context(), `UPDATE login_limits SET failures=$2,reset_at=CASE WHEN reset_at<=now() THEN now()+interval '15 minutes' ELSE reset_at END WHERE key=$1`, key, failures+1)
|
||||
if err == nil {
|
||||
err = tx.Commit(r.Context())
|
||||
}
|
||||
if err != nil {
|
||||
s.dbError(w, err)
|
||||
return
|
||||
}
|
||||
fail(w, 401, "invalid_credentials", "Invalid username or password")
|
||||
return
|
||||
}
|
||||
token := randomToken()
|
||||
a.CSRF = randomToken()
|
||||
_, err = tx.Exec(r.Context(), `DELETE FROM login_limits WHERE key=$1`, key)
|
||||
if err == nil {
|
||||
_, err = tx.Exec(r.Context(), `INSERT INTO sessions(token_hash,user_id,csrf_token,expires_at) VALUES($1,$2,$3,now()+interval '12 hours')`, hashToken(token), a.ID, a.CSRF)
|
||||
}
|
||||
if err == nil {
|
||||
err = tx.Commit(r.Context())
|
||||
}
|
||||
if err != nil {
|
||||
s.dbError(w, err)
|
||||
return
|
||||
}
|
||||
http.SetCookie(w, &http.Cookie{Name: "otche_session", Value: token, Path: "/api", HttpOnly: true, Secure: s.cfg.SecureCookies, SameSite: http.SameSiteStrictMode, MaxAge: 43200})
|
||||
s.session(w, r.WithContext(context.WithValue(r.Context(), authKey{}, a)))
|
||||
}
|
||||
func (s *Server) logout(w http.ResponseWriter, r *http.Request) {
|
||||
_, e := s.db.Exec(r.Context(), `DELETE FROM sessions WHERE token_hash=$1`, user(r).TokenHash)
|
||||
if e != nil {
|
||||
s.dbError(w, e)
|
||||
return
|
||||
}
|
||||
http.SetCookie(w, &http.Cookie{Name: "otche_session", Value: "", Path: "/api", HttpOnly: true, Secure: s.cfg.SecureCookies, SameSite: http.SameSiteStrictMode, MaxAge: -1})
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
func CreateUser(ctx context.Context, db *pgxpool.Pool, username, password, role string) (string, error) {
|
||||
if !usernameRE.MatchString(username) || len(password) < 14 || len(password) > 72 || strings.TrimSpace(password) != password || role != "admin" && role != "operator" {
|
||||
return "", errors.New("username must be 3-64 safe characters, password 14-72 bytes, role admin/operator")
|
||||
}
|
||||
h, e := bcrypt.GenerateFromPassword([]byte(password), 12)
|
||||
if e != nil {
|
||||
return "", e
|
||||
}
|
||||
id := store.NewID()
|
||||
_, e = db.Exec(ctx, `INSERT INTO users(id,username,password_hash,role)VALUES($1,$2,$3,$4)`, id, username, string(h), role)
|
||||
return id, e
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package grub
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const MaxPaths = 32
|
||||
|
||||
var device = regexp.MustCompile(`(?i)^(CON|PRN|AUX|NUL|COM[1-9¹²³]|LPT[1-9¹²³])(?:\.|$)`)
|
||||
|
||||
// Validate accepts exact local DOS file paths only; it never opens a host path.
|
||||
func Validate(paths []string) error {
|
||||
if len(paths) > MaxPaths {
|
||||
return errors.New("Grub accepts at most 32 file paths")
|
||||
}
|
||||
total := 0
|
||||
for _, p := range paths {
|
||||
total += len(p)
|
||||
if len(p) < 4 || len(p) > 1024 || total > 8192 || !((p[0] >= 'A' && p[0] <= 'Z') || (p[0] >= 'a' && p[0] <= 'z')) || p[1:3] != `:\` || strings.ContainsAny(p[3:], `/:*?"<>|`) {
|
||||
return errors.New("Grub requires bounded literal absolute local Windows file paths; devices, streams and wildcards are unsupported")
|
||||
}
|
||||
for _, r := range p {
|
||||
if r < 32 || r == 127 {
|
||||
return errors.New("Grub paths cannot contain control characters")
|
||||
}
|
||||
}
|
||||
for _, part := range strings.Split(p[3:], `\`) {
|
||||
if part == "" || part == "." || part == ".." || strings.HasSuffix(part, ".") || strings.HasSuffix(part, " ") || device.MatchString(part) {
|
||||
return errors.New("Grub path has an unsafe or directory component")
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type File struct {
|
||||
RequestedPath string `json:"requested_path"`
|
||||
ResolvedPath *string `json:"resolved_path"`
|
||||
Member *string `json:"member"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error"`
|
||||
Size *int64 `json:"size"`
|
||||
SHA256 *string `json:"sha256"`
|
||||
Snapshot *Snapshot `json:"snapshot"`
|
||||
}
|
||||
type Snapshot struct {
|
||||
OpenSize int64 `json:"open_size"`
|
||||
Changed bool `json:"changed"`
|
||||
Consistency string `json:"consistency"`
|
||||
}
|
||||
type Report struct {
|
||||
Status string `json:"status"`
|
||||
Requested int `json:"requested"`
|
||||
Collected int `json:"collected"`
|
||||
Files []File `json:"files"`
|
||||
}
|
||||
|
||||
// Member uses only a controller-assigned ordinal and sanitized basename.
|
||||
func Member(index int, path string) string {
|
||||
parts := strings.Split(path, `\`)
|
||||
name := parts[len(parts)-1]
|
||||
var b strings.Builder
|
||||
for _, r := range name {
|
||||
if r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '.' || r == '-' || r == '_' {
|
||||
b.WriteRune(r)
|
||||
} else {
|
||||
b.WriteByte('_')
|
||||
}
|
||||
if b.Len() >= 100 {
|
||||
break
|
||||
}
|
||||
}
|
||||
return fmt.Sprintf("files/%03d-%s", index+1, b.String())
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
package grub
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestLiteralLocalPaths(t *testing.T) {
|
||||
for _, path := range []string{`C:\Logs\app.log`, `D:\logs\100%done{copy}.log`, `C:\logs\no-extension`, `C:\Каталог\журнал.log`} {
|
||||
if err := Validate([]string{path}); err != nil {
|
||||
t.Fatalf("literal rejected %q: %v", path, err)
|
||||
}
|
||||
}
|
||||
for _, path := range []string{`\\server\share\file`, `\\?\C:\file`, `\\.\pipe\file`, `C:relative`, `C:\a\..\b`, `C:\a\x:stream`, `C:\a\*`, `C:\a\NUL`, `C:\a\COM1.txt`, `C:\a\LPT².log`, `C:\a\file.`, `C:\a\`, `%TEMP%\file`, "C:\\a\\x\x00"} {
|
||||
if err := Validate([]string{path}); err == nil {
|
||||
t.Fatalf("unsafe path accepted %q", path)
|
||||
}
|
||||
}
|
||||
if err := Validate(make([]string, 33)); err == nil {
|
||||
t.Fatal("count bound ignored")
|
||||
}
|
||||
paths := make([]string, 9)
|
||||
for i := range paths {
|
||||
paths[i] = `C:\` + strings.Repeat("x", 1000)
|
||||
}
|
||||
if err := Validate(paths); err == nil {
|
||||
t.Fatal("aggregate path bytes unbounded")
|
||||
}
|
||||
if err := Validate([]string{`C:\` + strings.Repeat("ж", 512)}); err == nil {
|
||||
t.Fatal("UTF8 byte limit ignored")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,360 @@
|
||||
package pve
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Client never includes response bodies, URLs or credentials in errors.
|
||||
type Client struct {
|
||||
base *url.URL
|
||||
http *http.Client
|
||||
auth string
|
||||
tls *tls.Config
|
||||
}
|
||||
type Credential struct {
|
||||
TokenID string `json:"token_id"`
|
||||
Secret string `json:"secret"`
|
||||
}
|
||||
type HTTPError struct{ Status int }
|
||||
|
||||
func (e *HTTPError) Error() string { return fmt.Sprintf("PVE HTTP status %d", e.Status) }
|
||||
func IsNotFound(err error) bool { var e *HTTPError; return errors.As(err, &e) && e.Status == 404 }
|
||||
func New(endpoint, caFile, credentialFile string) (*Client, error) {
|
||||
u, err := url.Parse(endpoint)
|
||||
if err != nil || u.Scheme != "https" || u.Host == "" || u.User != nil || u.RawQuery != "" || u.Fragment != "" {
|
||||
return nil, errors.New("PVE endpoint must be a credential-free HTTPS origin")
|
||||
}
|
||||
if u.Path != "" && u.Path != "/" {
|
||||
return nil, errors.New("PVE endpoint must not have a path")
|
||||
}
|
||||
b, err := os.ReadFile(credentialFile)
|
||||
if err != nil {
|
||||
return nil, errors.New("cannot read PVE credential file")
|
||||
}
|
||||
var cred Credential
|
||||
if err = json.Unmarshal(b, &cred); err != nil || !strings.Contains(cred.TokenID, "!") || cred.Secret == "" || strings.ContainsAny(cred.TokenID+cred.Secret, "\r\n") {
|
||||
return nil, errors.New("invalid PVE token file")
|
||||
}
|
||||
roots, err := x509.SystemCertPool()
|
||||
if err != nil {
|
||||
roots = x509.NewCertPool()
|
||||
}
|
||||
if caFile != "" {
|
||||
b, err = os.ReadFile(caFile)
|
||||
if err != nil || !roots.AppendCertsFromPEM(b) {
|
||||
return nil, errors.New("invalid PVE CA file")
|
||||
}
|
||||
}
|
||||
tc := &tls.Config{MinVersion: tls.VersionTLS12, RootCAs: roots}
|
||||
return &Client{base: u, http: &http.Client{Transport: &http.Transport{TLSClientConfig: tc, MaxIdleConnsPerHost: 8, ResponseHeaderTimeout: 30 * time.Second}, Timeout: 60 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error { return errors.New("PVE redirect refused") }}, auth: "PVEAPIToken=" + cred.TokenID + "=" + cred.Secret, tls: tc}, nil
|
||||
}
|
||||
func (c *Client) Close() { c.http.CloseIdleConnections() }
|
||||
func (c *Client) TLSConfig() *tls.Config { return c.tls.Clone() }
|
||||
func (c *Client) Header() http.Header { return http.Header{"Authorization": []string{c.auth}} }
|
||||
func (c *Client) endpoint(path string, q url.Values) string {
|
||||
u := *c.base
|
||||
escaped := "/api2/json" + path
|
||||
decoded, err := url.PathUnescape(escaped)
|
||||
if err != nil {
|
||||
decoded = escaped
|
||||
}
|
||||
u.Path = decoded
|
||||
u.RawPath = escaped
|
||||
u.RawQuery = q.Encode()
|
||||
return u.String()
|
||||
}
|
||||
func (c *Client) decode(req *http.Request, out any) error {
|
||||
req.Header.Set("Authorization", c.auth)
|
||||
resp, err := c.http.Do(req)
|
||||
if err != nil {
|
||||
if req.Context().Err() != nil {
|
||||
return req.Context().Err()
|
||||
}
|
||||
return errors.New("PVE transport failed")
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return &HTTPError{resp.StatusCode}
|
||||
}
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 24<<20+1))
|
||||
if err != nil {
|
||||
return errors.New("PVE response read failed")
|
||||
}
|
||||
if len(b) > 24<<20 {
|
||||
return errors.New("PVE response exceeded bound")
|
||||
}
|
||||
var envelope struct {
|
||||
Data json.RawMessage `json:"data"`
|
||||
}
|
||||
if json.Unmarshal(b, &envelope) != nil || len(envelope.Data) == 0 {
|
||||
return errors.New("malformed PVE envelope")
|
||||
}
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
if json.Unmarshal(envelope.Data, out) != nil {
|
||||
return errors.New("malformed PVE data")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (c *Client) Do(ctx context.Context, method, path string, values url.Values, out any) error {
|
||||
var body io.Reader
|
||||
query := url.Values{}
|
||||
if method == http.MethodGet || method == http.MethodDelete {
|
||||
query = values
|
||||
} else {
|
||||
body = strings.NewReader(values.Encode())
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, c.endpoint(path, query), body)
|
||||
if err != nil {
|
||||
return errors.New("invalid PVE request")
|
||||
}
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
}
|
||||
return c.decode(req, out)
|
||||
}
|
||||
|
||||
var upidRE = regexp.MustCompile(`^UPID:([A-Za-z0-9_.-]+):[A-Fa-f0-9]{8,16}:[A-Fa-f0-9]{8,16}:[A-Fa-f0-9]{8,16}:[A-Za-z0-9_-]+:[^:]*:[^:]+:$`)
|
||||
|
||||
func ValidateUPID(node, upid string) error {
|
||||
m := upidRE.FindStringSubmatch(upid)
|
||||
if m == nil || m[1] != node {
|
||||
return errors.New("malformed or wrong-node PVE UPID")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (c *Client) Task(ctx context.Context, node, upid string) error {
|
||||
if err := ValidateUPID(node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
ticker := time.NewTicker(time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
var s struct {
|
||||
Status string `json:"status"`
|
||||
ExitStatus string `json:"exitstatus"`
|
||||
}
|
||||
if err := c.Do(ctx, "GET", "/nodes/"+node+"/tasks/"+upid+"/status", nil, &s); err != nil {
|
||||
return err
|
||||
}
|
||||
switch s.Status {
|
||||
case "stopped":
|
||||
if s.ExitStatus != "OK" {
|
||||
return errors.New("PVE task failed")
|
||||
}
|
||||
return nil
|
||||
case "running":
|
||||
default:
|
||||
return errors.New("invalid PVE task status")
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
func VMPath(node string, vmid int) string { return "/nodes/" + node + "/qemu/" + strconv.Itoa(vmid) }
|
||||
|
||||
type Ownership struct {
|
||||
Node string
|
||||
VMID int
|
||||
Pool string
|
||||
OwnerID string
|
||||
AttemptID string
|
||||
Name string
|
||||
Protected []int
|
||||
}
|
||||
|
||||
func (o Ownership) Tags() string {
|
||||
return "otche;otche-owner-" + o.OwnerID + ";otche-attempt-" + o.AttemptID
|
||||
}
|
||||
func (c *Client) CheckOwned(ctx context.Context, o Ownership) error {
|
||||
if o.VMID < 100 || o.VMID == 7000 || o.VMID == 7001 || o.Pool == "" || o.OwnerID == "" || o.AttemptID == "" {
|
||||
return errors.New("protected or incomplete VM ownership")
|
||||
}
|
||||
for _, id := range o.Protected {
|
||||
if id == o.VMID {
|
||||
return errors.New("source VM is protected")
|
||||
}
|
||||
}
|
||||
var pool struct {
|
||||
Members []struct {
|
||||
VMID int `json:"vmid"`
|
||||
Node string `json:"node"`
|
||||
Type string `json:"type"`
|
||||
} `json:"members"`
|
||||
}
|
||||
if err := c.Do(ctx, "GET", "/pools/"+o.Pool, nil, &pool); err != nil {
|
||||
return err
|
||||
}
|
||||
found := false
|
||||
for _, v := range pool.Members {
|
||||
if v.VMID == o.VMID && v.Node == o.Node && v.Type == "qemu" {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return errors.New("VM is not in recorded owner pool")
|
||||
}
|
||||
cfg, err := c.Config(ctx, o.Node, o.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tags := map[string]bool{}
|
||||
for _, t := range strings.Split(Text(cfg["tags"]), ";") {
|
||||
tags[t] = true
|
||||
}
|
||||
for _, t := range strings.Split(o.Tags(), ";") {
|
||||
if !tags[t] {
|
||||
return errors.New("VM ownership tag mismatch")
|
||||
}
|
||||
}
|
||||
if Text(cfg["name"]) != o.Name {
|
||||
return errors.New("VM ownership name mismatch")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func Text(v any) string {
|
||||
if v == nil {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprint(v)
|
||||
}
|
||||
func (c *Client) Config(ctx context.Context, node string, id int) (map[string]any, error) {
|
||||
var v map[string]any
|
||||
err := c.Do(ctx, "GET", VMPath(node, id)+"/config", url.Values{"current": {"1"}}, &v)
|
||||
return v, err
|
||||
}
|
||||
func (c *Client) Status(ctx context.Context, node string, id int) (string, error) {
|
||||
var v struct {
|
||||
Status string `json:"status"`
|
||||
}
|
||||
err := c.Do(ctx, "GET", VMPath(node, id)+"/status/current", nil, &v)
|
||||
return v.Status, err
|
||||
}
|
||||
func (c *Client) NoPending(ctx context.Context, node string, id int, only string) error {
|
||||
var p []struct {
|
||||
Key string `json:"key"`
|
||||
Pending json.RawMessage `json:"pending"`
|
||||
Delete Bool `json:"delete"`
|
||||
}
|
||||
if err := c.Do(ctx, "GET", VMPath(node, id)+"/pending", nil, &p); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, v := range p {
|
||||
if only != "" && v.Key != only {
|
||||
continue
|
||||
}
|
||||
if (len(v.Pending) > 0 && string(v.Pending) != "null") || v.Delete {
|
||||
return errors.New("PVE pending configuration is not empty")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (c *Client) OwnedTask(ctx context.Context, o Ownership, method, suffix string, v url.Values) (string, error) {
|
||||
if err := c.CheckOwned(ctx, o); err != nil {
|
||||
return "", err
|
||||
}
|
||||
var upid string
|
||||
if err := c.Do(ctx, method, VMPath(o.Node, o.VMID)+suffix, v, &upid); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return upid, ValidateUPID(o.Node, upid)
|
||||
}
|
||||
func (c *Client) Upload(ctx context.Context, node, storage, filename, sha256 string, src io.ReadSeeker) (string, error) {
|
||||
start, err := src.Seek(0, io.SeekCurrent)
|
||||
if err != nil {
|
||||
return "", errors.New("cannot determine ISO upload offset")
|
||||
}
|
||||
end, err := src.Seek(0, io.SeekEnd)
|
||||
if err != nil {
|
||||
return "", errors.New("cannot determine ISO upload size")
|
||||
}
|
||||
if _, err = src.Seek(start, io.SeekStart); err != nil || end < start {
|
||||
return "", errors.New("cannot restore ISO upload offset")
|
||||
}
|
||||
var envelope bytes.Buffer
|
||||
mw := multipart.NewWriter(&envelope)
|
||||
if err = mw.WriteField("content", "iso"); err == nil {
|
||||
err = mw.WriteField("checksum-algorithm", "sha256")
|
||||
}
|
||||
if err == nil {
|
||||
err = mw.WriteField("checksum", sha256)
|
||||
}
|
||||
if err == nil {
|
||||
_, err = mw.CreateFormFile("filename", filename)
|
||||
}
|
||||
headerSize := envelope.Len()
|
||||
if err == nil {
|
||||
err = mw.Close()
|
||||
}
|
||||
if err != nil {
|
||||
return "", errors.New("cannot encode ISO upload envelope")
|
||||
}
|
||||
size := end - start
|
||||
if size > (1<<63-1)-int64(envelope.Len()) {
|
||||
return "", errors.New("ISO upload size exceeds bound")
|
||||
}
|
||||
// PVE rejects chunked uploads. Buffer only the envelope, never the ISO body.
|
||||
body := io.MultiReader(bytes.NewReader(envelope.Bytes()[:headerSize]), io.LimitReader(src, size), bytes.NewReader(envelope.Bytes()[headerSize:]))
|
||||
req, err := http.NewRequestWithContext(ctx, "POST", c.endpoint("/nodes/"+node+"/storage/"+storage+"/upload", nil), body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.ContentLength = int64(envelope.Len()) + size
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
var upid string
|
||||
// The upload has an explicit outer deadline; ordinary JSON calls retain their short timeout.
|
||||
hc := *c.http
|
||||
hc.Timeout = 0
|
||||
upload := *c
|
||||
upload.http = &hc
|
||||
if err = upload.decode(req, &upid); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return upid, ValidateUPID(node, upid)
|
||||
}
|
||||
func (c *Client) Console(ctx context.Context, o Ownership) (string, string, error) {
|
||||
// Recorder credential intentionally needs VM.Audit/Console only, not pool enumeration.
|
||||
if o.VMID < 100 || o.VMID == 7000 || o.VMID == 7001 {
|
||||
return "", "", errors.New("protected console")
|
||||
}
|
||||
for _, id := range o.Protected {
|
||||
if id == o.VMID {
|
||||
return "", "", errors.New("source console prohibited")
|
||||
}
|
||||
}
|
||||
var v struct {
|
||||
Ticket string `json:"ticket"`
|
||||
Password string `json:"password"`
|
||||
Port json.Number `json:"port"`
|
||||
}
|
||||
if err := c.Do(ctx, "POST", VMPath(o.Node, o.VMID)+"/vncproxy", url.Values{"websocket": {"1"}}, &v); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
if v.Ticket == "" || v.Password == "" || v.Port == "" {
|
||||
return "", "", errors.New("graphical console requires distinct WebSocket ticket and RFB password")
|
||||
}
|
||||
u := *c.base
|
||||
u.Scheme = "wss"
|
||||
u.Path = "/api2/json" + VMPath(o.Node, o.VMID) + "/vncwebsocket"
|
||||
u.RawQuery = url.Values{"port": {v.Port.String()}, "vncticket": {v.Ticket}}.Encode()
|
||||
return u.String(), v.Password, nil
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package pve
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func fixture(t *testing.T, h http.HandlerFunc) *Client {
|
||||
t.Helper()
|
||||
s := httptest.NewTLSServer(h)
|
||||
t.Cleanup(s.Close)
|
||||
u, err := url.Parse(s.URL)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return &Client{base: u, http: s.Client(), auth: "PVEAPIToken=fixture@pve!test=private", tls: s.Client().Transport.(*http.Transport).TLSClientConfig}
|
||||
}
|
||||
func data(w http.ResponseWriter, v any) { json.NewEncoder(w).Encode(map[string]any{"data": v}) }
|
||||
func TestOwnershipRefusesDestructiveForeignAndProtectedVM(t *testing.T) {
|
||||
o := Ownership{Node: "node", VMID: 9001, Pool: "owner", OwnerID: "alice", AttemptID: "attempt", Name: "otche-attempt"}
|
||||
mutations := 0
|
||||
tags := o.Tags()
|
||||
member := true
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != "GET" {
|
||||
mutations++
|
||||
data(w, "UPID:node:00000001:00000002:00000003:qmstop:9001:user@pve:")
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case strings.HasSuffix(r.URL.Path, "/pools/owner"):
|
||||
members := []map[string]any{}
|
||||
if member {
|
||||
members = append(members, map[string]any{"vmid": 9001, "node": "node", "type": "qemu"})
|
||||
}
|
||||
data(w, map[string]any{"members": members})
|
||||
case strings.HasSuffix(r.URL.Path, "/config"):
|
||||
data(w, map[string]any{"name": o.Name, "tags": tags})
|
||||
default:
|
||||
t.Errorf("unexpected request %s", r.URL.Path)
|
||||
}
|
||||
})
|
||||
tags = "otche;otche-owner-mallory;otche-attempt-attempt"
|
||||
if _, err := c.OwnedTask(context.Background(), o, "DELETE", "", nil); err == nil {
|
||||
t.Fatal("foreign tag accepted")
|
||||
}
|
||||
if mutations != 0 {
|
||||
t.Fatal("foreign VM was mutated")
|
||||
}
|
||||
tags = o.Tags()
|
||||
member = false
|
||||
if _, err := c.OwnedTask(context.Background(), o, "POST", "/status/stop", nil); err == nil {
|
||||
t.Fatal("wrong pool accepted")
|
||||
}
|
||||
member = true
|
||||
o.Protected = []int{9001}
|
||||
if _, err := c.OwnedTask(context.Background(), o, "DELETE", "", nil); err == nil {
|
||||
t.Fatal("protected source accepted")
|
||||
}
|
||||
if mutations != 0 {
|
||||
t.Fatal("protected VM was mutated")
|
||||
}
|
||||
o.Protected = nil
|
||||
if _, err := c.OwnedTask(context.Background(), o, "POST", "/status/stop", nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if mutations != 1 {
|
||||
t.Fatalf("owned mutation count %d", mutations)
|
||||
}
|
||||
}
|
||||
func TestMalformedUPIDNeverPolled(t *testing.T) {
|
||||
calls := 0
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
calls++
|
||||
data(w, map[string]any{"status": "stopped", "exitstatus": "OK"})
|
||||
})
|
||||
for _, upid := range []string{"OK", "UPID:other:00000001:00000002:00000003:qmclone:9001:user@pve:", "UPID:node:bad/../path", "UPID:node:00000001:00000002:00000003:qmclone:9001:user@pve"} {
|
||||
if err := c.Task(context.Background(), "node", upid); err == nil {
|
||||
t.Fatalf("accepted malformed UPID %q", upid)
|
||||
}
|
||||
}
|
||||
if calls != 0 {
|
||||
t.Fatal("malformed UPID generated HTTP requests")
|
||||
}
|
||||
}
|
||||
func TestTaskRequiresTerminalOK(t *testing.T) {
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
data(w, map[string]any{"status": "stopped", "exitstatus": "ERROR: cloning failed"})
|
||||
})
|
||||
if c.Task(context.Background(), "node", "UPID:node:00000001:00000002:00000003:qmclone:9001:user@pve:") == nil {
|
||||
t.Fatal("task failure treated as success")
|
||||
}
|
||||
}
|
||||
func TestQGAReadOffsetsUseDecodedBytesAndEnforceLimit(t *testing.T) {
|
||||
calls := 0
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
if q.Get("decode") != "0" {
|
||||
t.Error("binary decode was not disabled")
|
||||
}
|
||||
calls++
|
||||
switch calls {
|
||||
case 1:
|
||||
if q.Get("offset") != "0" {
|
||||
t.Error("first offset")
|
||||
}
|
||||
data(w, map[string]any{"content": base64.StdEncoding.EncodeToString([]byte{0, 255, 1}), "bytes-read": 3, "truncated": 1})
|
||||
case 2:
|
||||
if q.Get("offset") != "3" || q.Get("count") != "2" {
|
||||
t.Errorf("unexpected offset/count %v", q)
|
||||
}
|
||||
data(w, map[string]any{"content": base64.StdEncoding.EncodeToString([]byte{2, 3}), "bytes-read": 2, "truncated": false})
|
||||
default:
|
||||
t.Fatal("unbounded read")
|
||||
}
|
||||
})
|
||||
var out bytes.Buffer
|
||||
n, err := c.ReadFile(context.Background(), "node", 9001, "C:\\evidence.bin", &out, 5)
|
||||
if err != nil || n != 5 || !bytes.Equal(out.Bytes(), []byte{0, 255, 1, 2, 3}) {
|
||||
t.Fatalf("binary stream wrong: n=%d err=%v bytes=%v", n, err, out.Bytes())
|
||||
}
|
||||
}
|
||||
func TestQGARejectsOversizedAndNoProgressChunks(t *testing.T) {
|
||||
for _, bad := range []map[string]any{{"content": base64.StdEncoding.EncodeToString([]byte("too large")), "truncated": false}, {"content": "", "truncated": 1}} {
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) { data(w, bad) })
|
||||
var out bytes.Buffer
|
||||
if _, err := c.ReadFile(context.Background(), "node", 9001, "file", &out, 3); err == nil {
|
||||
t.Fatal("invalid QGA chunk accepted")
|
||||
}
|
||||
if out.Len() != 0 {
|
||||
t.Fatal("invalid chunk written before validation")
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestQGAControlWriteUsesPreencodedBytes(t *testing.T) {
|
||||
payload := []byte(`{"command_id":"abc"}`)
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
r.ParseForm()
|
||||
if r.Form.Get("encode") != "0" {
|
||||
t.Error("double-base64 write")
|
||||
}
|
||||
decoded, err := base64.StdEncoding.DecodeString(r.Form.Get("content"))
|
||||
if err != nil || !bytes.Equal(decoded, payload) {
|
||||
t.Error("control bytes changed")
|
||||
}
|
||||
data(w, nil)
|
||||
})
|
||||
if err := c.WriteControl(context.Background(), "node", 9001, "control.json", payload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
func TestTaskDeadlineStopsPolling(t *testing.T) {
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) { data(w, map[string]any{"status": "running"}) })
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
if c.Task(ctx, "node", "UPID:node:00000001:00000002:00000003:qmclone:9001:user@pve:") == nil {
|
||||
t.Fatal("deadline ignored")
|
||||
}
|
||||
}
|
||||
func TestConsoleSeparatesRFBPasswordFromTicket(t *testing.T) {
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
data(w, map[string]any{"ticket": "websocket-secret", "password": "rfb-secret", "port": 5901})
|
||||
})
|
||||
ws, password, err := c.Console(context.Background(), Ownership{Node: "node", VMID: 9001})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
u, _ := url.Parse(ws)
|
||||
if u.Query().Get("vncticket") != "websocket-secret" || password != "rfb-secret" {
|
||||
t.Fatal("console secrets conflated")
|
||||
}
|
||||
}
|
||||
func TestStorageVolumeRemainsSingleEscapedPathSegment(t *testing.T) {
|
||||
volume := "private:iso/otche-job.iso"
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if !strings.Contains(r.RequestURI, "/content/private:iso%2Fotche-job.iso") {
|
||||
t.Errorf("volume path was split or double escaped: %s", r.RequestURI)
|
||||
}
|
||||
data(w, nil)
|
||||
})
|
||||
if err := c.Do(context.Background(), "DELETE", "/nodes/node/storage/private/content/"+url.PathEscape(volume), nil, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadStreamsKnownLengthMultipartFromCurrentOffset(t *testing.T) {
|
||||
payload := []byte("ISO payload\x00\xff")
|
||||
const checksum = "ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb"
|
||||
const upid = "UPID:node:00000001:00000002:00000003:imgcopy::user@pve:"
|
||||
c := fixture(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost || r.URL.Path != "/api2/json/nodes/node/storage/private/upload" {
|
||||
t.Errorf("unexpected upload route: %s %s", r.Method, r.URL.Path)
|
||||
}
|
||||
if r.ContentLength <= 0 || len(r.TransferEncoding) != 0 {
|
||||
http.Error(w, "chunked uploads are not implemented", http.StatusNotImplemented)
|
||||
return
|
||||
}
|
||||
body, err := io.ReadAll(r.Body)
|
||||
if err != nil || int64(len(body)) != r.ContentLength {
|
||||
t.Errorf("upload body length: %d header: %d error: %v", len(body), r.ContentLength, err)
|
||||
http.Error(w, "incorrect body length", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
r.Body = io.NopCloser(bytes.NewReader(body))
|
||||
if err = r.ParseMultipartForm(1 << 20); err != nil {
|
||||
t.Error(err)
|
||||
http.Error(w, "invalid multipart", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
defer r.MultipartForm.RemoveAll()
|
||||
if r.FormValue("content") != "iso" || r.FormValue("checksum-algorithm") != "sha256" || r.FormValue("checksum") != checksum {
|
||||
t.Error("upload metadata changed")
|
||||
}
|
||||
file, header, err := r.FormFile("filename")
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return
|
||||
}
|
||||
defer file.Close()
|
||||
got, err := io.ReadAll(file)
|
||||
if err != nil || header.Filename != "fixture.iso" || !bytes.Equal(got, payload) {
|
||||
t.Errorf("uploaded file differs: name=%s body=%q error=%v", header.Filename, got, err)
|
||||
}
|
||||
data(w, upid)
|
||||
})
|
||||
source := bytes.NewReader(append([]byte("skip"), payload...))
|
||||
if _, err := source.Seek(4, io.SeekStart); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := c.Upload(context.Background(), "node", "private", "fixture.iso", checksum, source)
|
||||
if err != nil || got != upid {
|
||||
t.Fatalf("upload result=%q error=%v", got, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package pve
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
const FileChunk = 256 << 10
|
||||
const ControlLimit = 40 << 10
|
||||
|
||||
type Bool bool
|
||||
|
||||
func (b *Bool) UnmarshalJSON(v []byte) error {
|
||||
switch string(v) {
|
||||
case "true", "1":
|
||||
*b = true
|
||||
case "false", "0", "null":
|
||||
*b = false
|
||||
default:
|
||||
return errors.New("invalid PVE boolean")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ReadFile streams bounded binary chunks. Offsets advance by decoded bytes, never
|
||||
// by the requested count; QGA may return short reads with more data remaining.
|
||||
func (c *Client) ReadFile(ctx context.Context, node string, id int, path string, dst io.Writer, limit int64) (int64, error) {
|
||||
if limit < 1 || limit > 1<<30 {
|
||||
return 0, errors.New("invalid guest read bound")
|
||||
}
|
||||
var offset int64
|
||||
for {
|
||||
count := int64(FileChunk)
|
||||
if limit-offset < count {
|
||||
count = limit - offset
|
||||
}
|
||||
if count == 0 {
|
||||
return offset, errors.New("guest file exceeds read limit")
|
||||
}
|
||||
var r struct {
|
||||
Content string `json:"content"`
|
||||
Truncated Bool `json:"truncated"`
|
||||
BytesRead *int64 `json:"bytes-read"`
|
||||
}
|
||||
err := c.Do(ctx, "GET", VMPath(node, id)+"/agent/file-read", url.Values{"file": {path}, "offset": {strconv.FormatInt(offset, 10)}, "count": {strconv.FormatInt(count, 10)}, "decode": {"0"}}, &r)
|
||||
if err != nil {
|
||||
return offset, err
|
||||
}
|
||||
b, err := base64.StdEncoding.DecodeString(r.Content)
|
||||
if err != nil || int64(len(b)) > count || (r.BytesRead != nil && *r.BytesRead != int64(len(b))) {
|
||||
return offset, errors.New("malformed or oversized QGA file chunk")
|
||||
}
|
||||
if len(b) == 0 && r.Truncated {
|
||||
return offset, errors.New("QGA file read made no progress")
|
||||
}
|
||||
n, err := dst.Write(b)
|
||||
offset += int64(n)
|
||||
if err != nil {
|
||||
return offset, err
|
||||
}
|
||||
if n != len(b) {
|
||||
return offset, io.ErrShortWrite
|
||||
}
|
||||
if !r.Truncated {
|
||||
return offset, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
func (c *Client) WriteControl(ctx context.Context, node string, id int, path string, b []byte) error {
|
||||
if len(b) == 0 || len(b) > ControlLimit || !json.Valid(b) {
|
||||
return errors.New("invalid bounded control JSON")
|
||||
}
|
||||
return c.Do(ctx, "POST", VMPath(node, id)+"/agent/file-write", url.Values{"file": {path}, "content": {base64.StdEncoding.EncodeToString(b)}, "encode": {"0"}}, nil)
|
||||
}
|
||||
func (c *Client) Exec(ctx context.Context, node string, id int, args []string) (int, error) {
|
||||
if len(args) == 0 || len(args) > 16 {
|
||||
return 0, errors.New("invalid QGA control command")
|
||||
}
|
||||
v := url.Values{}
|
||||
total := 0
|
||||
for _, a := range args {
|
||||
total += len(a)
|
||||
v.Add("command", a)
|
||||
}
|
||||
if total > 16384 {
|
||||
return 0, errors.New("QGA command exceeds control bound")
|
||||
}
|
||||
var r struct {
|
||||
PID int `json:"pid"`
|
||||
}
|
||||
if err := c.Do(ctx, "POST", VMPath(node, id)+"/agent/exec", v, &r); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if r.PID <= 0 {
|
||||
return 0, errors.New("invalid QGA execution pid")
|
||||
}
|
||||
return r.PID, nil
|
||||
}
|
||||
func (c *Client) ExecWait(ctx context.Context, node string, id, pid int) ([]byte, error) {
|
||||
t := time.NewTicker(time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
var r struct {
|
||||
Exited any `json:"exited"`
|
||||
ExitCode int `json:"exitcode"`
|
||||
Out string `json:"out-data"`
|
||||
Truncated Bool `json:"out-truncated"`
|
||||
}
|
||||
if err := c.Do(ctx, "GET", VMPath(node, id)+"/agent/exec-status", url.Values{"pid": {strconv.Itoa(pid)}}, &r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if Text(r.Exited) == "1" || Text(r.Exited) == "true" {
|
||||
if r.Truncated || len(r.Out) > ControlLimit {
|
||||
return nil, errors.New("QGA control output exceeded bound")
|
||||
}
|
||||
if r.ExitCode != 0 {
|
||||
return []byte(r.Out), errors.New("guest control command failed")
|
||||
}
|
||||
return []byte(r.Out), nil
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case <-t.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
package recorder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
const encoderTimeout = 10 * time.Second
|
||||
|
||||
// fragmentSink writes only inside the worker's private recording directory. Its
|
||||
// scanner accepts bounded, complete ISO-BMFF boxes, and readiness means an mdat
|
||||
// following moof has actually reached a synced file, not merely an open pipe.
|
||||
type fragmentSink struct {
|
||||
file *os.File
|
||||
remaining *int64
|
||||
bytes int64
|
||||
header [16]byte
|
||||
headerBytes int
|
||||
payload uint64
|
||||
kind string
|
||||
sawType, sawMovie, sawFragment bool
|
||||
fragments int
|
||||
ready chan struct{}
|
||||
err error
|
||||
}
|
||||
|
||||
func (s *fragmentSink) Write(p []byte) (int, error) {
|
||||
if s.err != nil {
|
||||
return 0, s.err
|
||||
}
|
||||
if int64(len(p)) > *s.remaining {
|
||||
s.err = errors.New("recording byte limit exceeded")
|
||||
return 0, s.err
|
||||
}
|
||||
n, err := s.file.Write(p)
|
||||
s.bytes += int64(n)
|
||||
*s.remaining -= int64(n)
|
||||
if err == nil && n != len(p) {
|
||||
err = io.ErrShortWrite
|
||||
}
|
||||
if err == nil {
|
||||
err = s.scan(p[:n])
|
||||
}
|
||||
if err != nil {
|
||||
s.err = err
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
func (s *fragmentSink) scan(p []byte) error {
|
||||
for len(p) > 0 {
|
||||
if s.payload > 0 {
|
||||
n := min(uint64(len(p)), s.payload)
|
||||
p = p[n:]
|
||||
s.payload -= n
|
||||
if s.payload == 0 {
|
||||
if err := s.endBox(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
need := 8
|
||||
if s.headerBytes >= 8 && binary.BigEndian.Uint32(s.header[:4]) == 1 {
|
||||
need = 16
|
||||
}
|
||||
n := min(len(p), need-s.headerBytes)
|
||||
copy(s.header[s.headerBytes:], p[:n])
|
||||
s.headerBytes += n
|
||||
p = p[n:]
|
||||
if s.headerBytes < need {
|
||||
continue
|
||||
}
|
||||
size := uint64(binary.BigEndian.Uint32(s.header[:4]))
|
||||
if size == 1 {
|
||||
if s.headerBytes < 16 {
|
||||
continue
|
||||
}
|
||||
size = binary.BigEndian.Uint64(s.header[8:16])
|
||||
}
|
||||
if size < uint64(need) || size > maxMessageBytes {
|
||||
return errors.New("invalid encoded MP4 box")
|
||||
}
|
||||
s.kind = string(s.header[4:8])
|
||||
s.headerBytes = 0
|
||||
s.payload = size - uint64(need)
|
||||
if s.payload == 0 {
|
||||
if err := s.endBox(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *fragmentSink) endBox() error {
|
||||
switch s.kind {
|
||||
case "ftyp":
|
||||
s.sawType = true
|
||||
case "moov":
|
||||
s.sawMovie = true
|
||||
case "moof":
|
||||
s.sawFragment = true
|
||||
case "mdat":
|
||||
if !s.sawType || !s.sawMovie || !s.sawFragment {
|
||||
return errors.New("encoded MP4 fragment has no initialization")
|
||||
}
|
||||
if err := s.file.Sync(); err != nil {
|
||||
return errors.New("recording sink sync failed")
|
||||
}
|
||||
s.fragments++
|
||||
s.sawFragment = false
|
||||
if s.fragments == 1 {
|
||||
close(s.ready)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type encoder struct {
|
||||
stdin io.WriteCloser
|
||||
sink *fragmentSink
|
||||
done chan struct{}
|
||||
waitErr error
|
||||
cancel context.CancelFunc
|
||||
closeOnce sync.Once
|
||||
frames int64
|
||||
}
|
||||
|
||||
func startEncoder(binary string, root *os.Root, name string, width, height, fps int, remaining *int64) (*encoder, error) {
|
||||
file, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return nil, errors.New("cannot create private recording segment")
|
||||
}
|
||||
if err := syncDirectory(root); err != nil {
|
||||
file.Close()
|
||||
return nil, err
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
args := []string{
|
||||
"-hide_banner", "-loglevel", "error", "-nostdin", "-probesize", "32", "-analyzeduration", "0",
|
||||
"-f", "rawvideo", "-pixel_format", "rgb24", "-video_size", fmt.Sprintf("%dx%d", width, height),
|
||||
"-framerate", strconv.Itoa(fps), "-i", "pipe:0", "-an", "-c:v", "libx264",
|
||||
"-preset", "ultrafast", "-tune", "zerolatency", "-threads", "1", "-filter_threads", "1",
|
||||
"-vf", "pad=ceil(iw/2)*2:ceil(ih/2)*2", "-pix_fmt", "yuv420p",
|
||||
"-g", strconv.Itoa(fps * 2), "-bf", "0", "-f", "mp4",
|
||||
"-movflags", "+empty_moov+default_base_moof+frag_every_frame", "-flush_packets", "1", "pipe:1",
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, binary, args...)
|
||||
cmd.WaitDelay = 2 * time.Second
|
||||
sink := &fragmentSink{file: file, remaining: remaining, ready: make(chan struct{})}
|
||||
cmd.Stdout = sink
|
||||
// Peer data, tickets and paths never enter the command line or diagnostics.
|
||||
// Encoder stderr is not forwarded to application logs.
|
||||
cmd.Stderr = io.Discard
|
||||
stdin, err := cmd.StdinPipe()
|
||||
if err != nil {
|
||||
cancel()
|
||||
file.Close()
|
||||
return nil, errors.New("cannot open FFmpeg frame input")
|
||||
}
|
||||
e := &encoder{stdin: stdin, sink: sink, done: make(chan struct{}), cancel: cancel}
|
||||
if err := cmd.Start(); err != nil {
|
||||
cancel()
|
||||
stdin.Close()
|
||||
file.Close()
|
||||
return nil, errors.New("cannot start FFmpeg H264 encoder")
|
||||
}
|
||||
go func() { e.waitErr = cmd.Wait(); close(e.done) }()
|
||||
return e, nil
|
||||
}
|
||||
|
||||
func (e *encoder) writeFrame(pixels []byte) error {
|
||||
timer := time.AfterFunc(encoderTimeout, e.cancel)
|
||||
defer timer.Stop()
|
||||
for len(pixels) > 0 {
|
||||
n, err := e.stdin.Write(pixels)
|
||||
if err != nil {
|
||||
return errors.New("FFmpeg frame input failed or timed out")
|
||||
}
|
||||
if n == 0 {
|
||||
return errors.New("FFmpeg frame input made no progress")
|
||||
}
|
||||
pixels = pixels[n:]
|
||||
}
|
||||
e.frames++
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e *encoder) finish() error {
|
||||
e.closeOnce.Do(func() { e.stdin.Close() })
|
||||
timer := time.NewTimer(encoderTimeout)
|
||||
defer timer.Stop()
|
||||
select {
|
||||
case <-e.done:
|
||||
case <-timer.C:
|
||||
e.cancel()
|
||||
<-e.done
|
||||
}
|
||||
defer e.cancel()
|
||||
syncErr := e.sink.file.Sync()
|
||||
closeErr := e.sink.file.Close()
|
||||
if e.sink.err != nil {
|
||||
return e.sink.err
|
||||
}
|
||||
if e.waitErr != nil {
|
||||
return errors.New("FFmpeg did not finish successfully")
|
||||
}
|
||||
if syncErr != nil || closeErr != nil {
|
||||
return errors.New("recording sink finalization failed")
|
||||
}
|
||||
if e.frames == 0 || e.sink.fragments == 0 || e.sink.payload != 0 || e.sink.headerBytes != 0 || e.sink.sawFragment {
|
||||
return errors.New("recording segment is incomplete")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,491 @@
|
||||
// Package recorder is a read-only RFB viewer with a private, durable H264 sink.
|
||||
// It never sends keyboard, pointer, clipboard or other guest input events.
|
||||
package recorder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/gorilla/websocket"
|
||||
)
|
||||
|
||||
const (
|
||||
manifestAllowance = 128 << 10
|
||||
maxSegments = 256
|
||||
segmentDuration = 60 * time.Second
|
||||
)
|
||||
|
||||
var errFreshness = errors.New("RFB full-frame freshness deadline exceeded")
|
||||
|
||||
type Config struct {
|
||||
URL string
|
||||
Header http.Header
|
||||
TLSConfig *tls.Config
|
||||
Ticket string
|
||||
Directory string
|
||||
FPS int
|
||||
MaxBytes int64
|
||||
}
|
||||
|
||||
type Segment struct {
|
||||
Filename string `json:"filename"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
FinishedAt *time.Time `json:"finished_at"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
FPS int `json:"fps"`
|
||||
Frames int64 `json:"frames"`
|
||||
Size int64 `json:"size"`
|
||||
State string `json:"state"`
|
||||
}
|
||||
|
||||
type Gap struct {
|
||||
At time.Time `json:"at"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
type Manifest struct {
|
||||
State string `json:"state"`
|
||||
StartedAt *time.Time `json:"started_at"`
|
||||
FinishedAt *time.Time `json:"finished_at"`
|
||||
Segments []Segment `json:"segments"`
|
||||
Gaps []Gap `json:"gaps"`
|
||||
}
|
||||
|
||||
type Session struct {
|
||||
ctx context.Context
|
||||
parent context.Context
|
||||
cancel context.CancelFunc
|
||||
conn *websocket.Conn
|
||||
frame *framebuffer
|
||||
root *os.Root
|
||||
directory string
|
||||
ffmpeg string
|
||||
fps int
|
||||
remaining int64
|
||||
manifest Manifest
|
||||
done chan struct{}
|
||||
ready chan struct{}
|
||||
closing atomic.Bool
|
||||
mu sync.Mutex
|
||||
err error
|
||||
}
|
||||
|
||||
// Start returns only after a complete, real framebuffer has been decoded and
|
||||
// the first encoded MP4 fragment has been synced to the private recording sink.
|
||||
// The caller must monitor Done even after Start succeeds. Cancellation is a
|
||||
// partial recording; only an explicit Close can produce state complete.
|
||||
func Start(ctx context.Context, config Config) (*Session, error) {
|
||||
if ctx == nil {
|
||||
return nil, errors.New("recording context required")
|
||||
}
|
||||
if config.FPS == 0 {
|
||||
config.FPS = 5
|
||||
}
|
||||
if config.FPS < 1 || config.FPS > 15 {
|
||||
return nil, errors.New("recording FPS must be between 1 and 15")
|
||||
}
|
||||
if config.MaxBytes == 0 {
|
||||
config.MaxBytes = 1 << 30
|
||||
}
|
||||
if config.MaxBytes < 1<<20 {
|
||||
return nil, errors.New("recording byte limit must be at least 1 MiB")
|
||||
}
|
||||
u, err := url.Parse(config.URL)
|
||||
if err != nil || u.Host == "" || (u.Scheme != "ws" && u.Scheme != "wss") || u.User != nil || u.Fragment != "" {
|
||||
return nil, errors.New("invalid recording WebSocket URL")
|
||||
}
|
||||
if config.Directory == "" {
|
||||
return nil, errors.New("private recording directory required")
|
||||
}
|
||||
binary, err := exec.LookPath("ffmpeg")
|
||||
if err != nil {
|
||||
return nil, errors.New("FFmpeg with libx264 is required on worker PATH")
|
||||
}
|
||||
directory, err := filepath.Abs(config.Directory)
|
||||
if err != nil {
|
||||
return nil, errors.New("invalid private recording directory")
|
||||
}
|
||||
if err := os.MkdirAll(directory, 0700); err != nil {
|
||||
return nil, errors.New("cannot create private recording directory")
|
||||
}
|
||||
info, err := os.Lstat(directory)
|
||||
if err != nil || !info.IsDir() || info.Mode()&os.ModeSymlink != 0 {
|
||||
return nil, errors.New("recording directory must not be a symlink")
|
||||
}
|
||||
if err := os.Chmod(directory, 0700); err != nil {
|
||||
return nil, errors.New("cannot protect private recording directory")
|
||||
}
|
||||
root, err := os.OpenRoot(directory)
|
||||
if err != nil {
|
||||
return nil, errors.New("cannot open private recording directory")
|
||||
}
|
||||
dir, err := root.Open(".")
|
||||
if err != nil {
|
||||
root.Close()
|
||||
return nil, errors.New("cannot inspect recording directory")
|
||||
}
|
||||
entries, readErr := dir.Readdirnames(1)
|
||||
dir.Close()
|
||||
if len(entries) != 0 || (readErr != nil && !errors.Is(readErr, io.EOF)) {
|
||||
root.Close()
|
||||
return nil, errors.New("recording directory must be empty; existing evidence is never overwritten")
|
||||
}
|
||||
dialer := websocket.Dialer{HandshakeTimeout: 15 * time.Second, Subprotocols: []string{"binary"}, ReadBufferSize: 32 << 10, WriteBufferSize: 1024}
|
||||
if config.TLSConfig != nil {
|
||||
dialer.TLSClientConfig = config.TLSConfig.Clone()
|
||||
}
|
||||
connectCtx, connectCancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
conn, response, err := dialer.DialContext(connectCtx, config.URL, config.Header.Clone())
|
||||
if response != nil && response.Body != nil {
|
||||
response.Body.Close()
|
||||
}
|
||||
if err != nil {
|
||||
connectCancel()
|
||||
root.Close()
|
||||
return nil, errors.New("recording WebSocket connection failed")
|
||||
}
|
||||
conn.SetReadLimit(maxMessageBytes)
|
||||
conn.EnableWriteCompression(false)
|
||||
conn.SetReadDeadline(time.Now().Add(20 * time.Second))
|
||||
stopHandshake := context.AfterFunc(connectCtx, func() { conn.Close() })
|
||||
frame, err := negotiate(&wsStream{conn: conn}, config.Ticket)
|
||||
stopped := stopHandshake()
|
||||
connectCancel()
|
||||
if err != nil || !stopped {
|
||||
conn.Close()
|
||||
root.Close()
|
||||
if err == nil {
|
||||
err = errors.New("RFB negotiation cancelled or timed out")
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if err := conn.SetReadDeadline(time.Time{}); err != nil {
|
||||
conn.Close()
|
||||
root.Close()
|
||||
return nil, errors.New("RFB deadline setup failed")
|
||||
}
|
||||
runCtx, cancel := context.WithCancel(ctx)
|
||||
s := &Session{
|
||||
ctx: runCtx, parent: ctx, cancel: cancel, conn: conn, frame: frame, root: root,
|
||||
directory: directory, ffmpeg: binary, fps: config.FPS,
|
||||
remaining: config.MaxBytes - 2*manifestAllowance,
|
||||
manifest: Manifest{State: "unavailable", Segments: []Segment{}, Gaps: []Gap{}},
|
||||
done: make(chan struct{}), ready: make(chan struct{}),
|
||||
}
|
||||
go s.run()
|
||||
select {
|
||||
case <-s.ready:
|
||||
if err := s.Err(); err != nil {
|
||||
<-s.done
|
||||
return nil, err
|
||||
}
|
||||
return s, nil
|
||||
case <-s.done:
|
||||
return nil, s.Err()
|
||||
case <-ctx.Done():
|
||||
<-s.done
|
||||
return nil, s.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Session) Done() <-chan struct{} { return s.done }
|
||||
|
||||
func (s *Session) Err() error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.err
|
||||
}
|
||||
|
||||
func (s *Session) setError(err error) {
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
if s.err == nil {
|
||||
s.err = err
|
||||
}
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *Session) fail(err error) {
|
||||
s.setError(err)
|
||||
s.cancel()
|
||||
s.conn.Close()
|
||||
}
|
||||
|
||||
// Close finalizes and validates the current fragment stream without deleting
|
||||
// anything. It is idempotent; a previous recording failure remains an error.
|
||||
func (s *Session) Close() error {
|
||||
select {
|
||||
case <-s.done:
|
||||
return s.Err()
|
||||
default:
|
||||
}
|
||||
if s.parent.Err() != nil {
|
||||
s.setError(errors.New("recording context cancelled or deadline exceeded"))
|
||||
}
|
||||
s.closing.Store(true)
|
||||
s.cancel()
|
||||
s.conn.Close()
|
||||
<-s.done
|
||||
return s.Err()
|
||||
}
|
||||
|
||||
func (s *Session) saveManifest() error {
|
||||
data, err := json.Marshal(s.manifest)
|
||||
if err != nil || len(data) > manifestAllowance {
|
||||
return errors.New("recording manifest exceeds limit")
|
||||
}
|
||||
file, err := s.root.OpenFile("manifest.next", os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
|
||||
if err != nil {
|
||||
return errors.New("cannot create recording manifest")
|
||||
}
|
||||
_, writeErr := file.Write(data)
|
||||
if writeErr == nil {
|
||||
writeErr = file.Sync()
|
||||
}
|
||||
closeErr := file.Close()
|
||||
if writeErr != nil || closeErr != nil {
|
||||
s.root.Remove("manifest.next")
|
||||
return errors.New("cannot persist recording manifest")
|
||||
}
|
||||
// Both generated names belong to an empty, mode-0700 directory. No remote
|
||||
// paths, filenames or desktop names ever participate in filesystem access.
|
||||
if err := os.Rename(filepath.Join(s.directory, "manifest.next"), filepath.Join(s.directory, "manifest.json")); err != nil {
|
||||
s.root.Remove("manifest.next")
|
||||
return errors.New("cannot publish recording manifest")
|
||||
}
|
||||
return syncDirectory(s.root)
|
||||
}
|
||||
|
||||
func syncDirectory(root *os.Root) error {
|
||||
// Windows flushes each output file; directory handles do not support Sync.
|
||||
if runtime.GOOS == "windows" {
|
||||
return nil
|
||||
}
|
||||
directory, err := root.Open(".")
|
||||
if err != nil {
|
||||
return errors.New("cannot open recording directory for sync")
|
||||
}
|
||||
defer directory.Close()
|
||||
if err := directory.Sync(); err != nil {
|
||||
return errors.New("cannot sync recording directory")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Session) run() {
|
||||
readerDone := make(chan struct{})
|
||||
go func() {
|
||||
defer close(readerDone)
|
||||
pace := time.NewTicker(time.Second / time.Duration(s.fps))
|
||||
defer pace.Stop()
|
||||
for {
|
||||
if err := s.frame.readMessage(); err != nil {
|
||||
if s.ctx.Err() == nil {
|
||||
s.fail(err)
|
||||
}
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-s.ctx.Done():
|
||||
return
|
||||
case <-pace.C:
|
||||
}
|
||||
}
|
||||
}()
|
||||
watcherDone := make(chan struct{})
|
||||
go func() {
|
||||
defer close(watcherDone)
|
||||
ticker := time.NewTicker(250 * time.Millisecond)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-s.ctx.Done():
|
||||
s.conn.Close()
|
||||
return
|
||||
case <-ticker.C:
|
||||
if time.Since(time.Unix(0, s.frame.lastFull.Load())) > freshnessLimit {
|
||||
s.fail(errFreshness)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
var current *encoder
|
||||
var generation uint64
|
||||
announced := false
|
||||
finishSegment := func() {
|
||||
if current == nil {
|
||||
return
|
||||
}
|
||||
err := current.finish()
|
||||
segment := &s.manifest.Segments[len(s.manifest.Segments)-1]
|
||||
now := time.Now().UTC()
|
||||
segment.FinishedAt = &now
|
||||
segment.Frames, segment.Size = current.frames, current.sink.bytes
|
||||
segment.State = "complete"
|
||||
if err != nil {
|
||||
segment.State = "partial"
|
||||
s.fail(err)
|
||||
}
|
||||
current = nil
|
||||
}
|
||||
defer func() {
|
||||
s.cancel()
|
||||
s.conn.Close()
|
||||
<-readerDone
|
||||
<-watcherDone
|
||||
finishSegment()
|
||||
if !s.closing.Load() && s.Err() == nil {
|
||||
s.setError(errors.New("recording context cancelled or deadline exceeded"))
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
s.manifest.FinishedAt = &now
|
||||
s.manifest.State = "complete"
|
||||
if err := s.Err(); err != nil {
|
||||
s.manifest.State = "partial"
|
||||
if s.manifest.StartedAt == nil {
|
||||
s.manifest.State = "unavailable"
|
||||
}
|
||||
at := now
|
||||
if errors.Is(err, errFreshness) {
|
||||
at = time.Unix(0, s.frame.lastFull.Load()).UTC()
|
||||
}
|
||||
s.manifest.Gaps = append(s.manifest.Gaps, Gap{At: at, Reason: err.Error()})
|
||||
}
|
||||
if err := s.saveManifest(); err != nil {
|
||||
s.setError(err)
|
||||
}
|
||||
s.root.Close()
|
||||
close(s.done)
|
||||
}()
|
||||
if err := s.saveManifest(); err != nil {
|
||||
s.fail(err)
|
||||
return
|
||||
}
|
||||
if err := s.frame.request(true); err != nil {
|
||||
s.fail(err)
|
||||
return
|
||||
}
|
||||
ticker := time.NewTicker(time.Second / time.Duration(s.fps))
|
||||
defer ticker.Stop()
|
||||
lastFullRequest := time.Now()
|
||||
var lastUpdate uint64
|
||||
ready := false
|
||||
for {
|
||||
select {
|
||||
case <-s.ctx.Done():
|
||||
return
|
||||
case tick := <-ticker.C:
|
||||
if current != nil {
|
||||
select {
|
||||
case <-current.done:
|
||||
s.fail(errors.New("FFmpeg encoder stopped during recording"))
|
||||
return
|
||||
default:
|
||||
}
|
||||
}
|
||||
full := tick.Sub(lastFullRequest) >= fullUpdateInterval
|
||||
update := s.frame.updates.Load()
|
||||
if full || update != lastUpdate {
|
||||
if err := s.frame.request(full); err != nil {
|
||||
if s.ctx.Err() == nil {
|
||||
s.fail(err)
|
||||
}
|
||||
return
|
||||
}
|
||||
lastUpdate = update
|
||||
if full {
|
||||
lastFullRequest = tick
|
||||
}
|
||||
}
|
||||
s.frame.mu.Lock()
|
||||
resized := current != nil && generation != s.frame.generation
|
||||
rotate := current != nil && tick.Sub(s.manifest.Segments[len(s.manifest.Segments)-1].StartedAt) >= segmentDuration
|
||||
if resized || rotate {
|
||||
s.frame.mu.Unlock()
|
||||
finishSegment()
|
||||
if resized {
|
||||
s.manifest.Gaps = append(s.manifest.Gaps, Gap{At: tick.UTC(), Reason: "framebuffer resized; awaiting complete new frame"})
|
||||
}
|
||||
if err := s.saveManifest(); err != nil {
|
||||
s.fail(err)
|
||||
return
|
||||
}
|
||||
if s.Err() != nil {
|
||||
return
|
||||
}
|
||||
s.frame.mu.Lock()
|
||||
}
|
||||
if !s.frame.complete {
|
||||
s.frame.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
newSegment := current == nil
|
||||
if newSegment {
|
||||
if len(s.manifest.Segments) >= maxSegments {
|
||||
s.frame.mu.Unlock()
|
||||
s.fail(errors.New("recording segment limit exceeded"))
|
||||
return
|
||||
}
|
||||
name := fmt.Sprintf("segment-%06d.mp4", len(s.manifest.Segments)+1)
|
||||
var err error
|
||||
current, err = startEncoder(s.ffmpeg, s.root, name, s.frame.width, s.frame.height, s.fps, &s.remaining)
|
||||
if err != nil {
|
||||
s.frame.mu.Unlock()
|
||||
s.fail(err)
|
||||
return
|
||||
}
|
||||
generation = s.frame.generation
|
||||
announced = false
|
||||
s.manifest.Segments = append(s.manifest.Segments, Segment{Filename: name, StartedAt: tick.UTC(), Width: s.frame.width, Height: s.frame.height, FPS: s.fps, State: "recording"})
|
||||
}
|
||||
err := current.writeFrame(s.frame.pixels)
|
||||
s.frame.mu.Unlock()
|
||||
if err != nil {
|
||||
s.fail(err)
|
||||
return
|
||||
}
|
||||
if !announced {
|
||||
select {
|
||||
case <-current.sink.ready:
|
||||
announced = true
|
||||
default:
|
||||
if tick.Sub(s.manifest.Segments[len(s.manifest.Segments)-1].StartedAt) > encoderTimeout {
|
||||
s.fail(errors.New("FFmpeg produced no durable video fragment"))
|
||||
return
|
||||
}
|
||||
continue
|
||||
}
|
||||
if s.manifest.StartedAt == nil {
|
||||
start := s.manifest.Segments[0].StartedAt
|
||||
s.manifest.StartedAt = &start
|
||||
}
|
||||
s.manifest.State = "recording"
|
||||
if err := s.saveManifest(); err != nil {
|
||||
s.fail(err)
|
||||
return
|
||||
}
|
||||
if !ready {
|
||||
close(s.ready)
|
||||
ready = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,565 @@
|
||||
package recorder
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/des"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/gorilla/websocket"
|
||||
)
|
||||
|
||||
func fixture(t *testing.T, handle func(*wsStream) error) (string, <-chan error) {
|
||||
t.Helper()
|
||||
result := make(chan error, 1)
|
||||
upgrader := websocket.Upgrader{Subprotocols: []string{"binary"}}
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
conn, err := upgrader.Upgrade(w, r, nil)
|
||||
if err != nil {
|
||||
result <- err
|
||||
return
|
||||
}
|
||||
defer conn.Close()
|
||||
conn.SetReadDeadline(time.Now().Add(30 * time.Second))
|
||||
result <- handle(&wsStream{conn: conn})
|
||||
}))
|
||||
t.Cleanup(server.Close)
|
||||
return "ws" + strings.TrimPrefix(server.URL, "http"), result
|
||||
}
|
||||
|
||||
func sendSplit(stream *wsStream, p []byte) error {
|
||||
for len(p) > 0 {
|
||||
n := min(7, len(p))
|
||||
if err := stream.write(p[:n]); err != nil {
|
||||
return err
|
||||
}
|
||||
p = p[n:]
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func serverHandshake(stream *wsStream, minor, width, height int, auth bool) error {
|
||||
version := []byte(fmt.Sprintf("RFB 003.%03d\n", minor))
|
||||
if err := sendSplit(stream, version); err != nil {
|
||||
return err
|
||||
}
|
||||
answer := make([]byte, 12)
|
||||
if _, err := io.ReadFull(stream, answer); err != nil {
|
||||
return err
|
||||
}
|
||||
if !bytes.Equal(answer, version) {
|
||||
return errors.New("wrong protocol negotiation")
|
||||
}
|
||||
security := byte(1)
|
||||
if auth {
|
||||
security = 2
|
||||
}
|
||||
if minor == 3 {
|
||||
if err := stream.write([]byte{0, 0, 0, security}); err != nil {
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
if err := stream.write([]byte{1, security}); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.ReadFull(stream, answer[:1]); err != nil {
|
||||
return err
|
||||
}
|
||||
if answer[0] != security {
|
||||
return errors.New("wrong security selection")
|
||||
}
|
||||
}
|
||||
if auth {
|
||||
challenge := []byte{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}
|
||||
if err := sendSplit(stream, challenge); err != nil {
|
||||
return err
|
||||
}
|
||||
response := make([]byte, 16)
|
||||
if _, err := io.ReadFull(stream, response); err != nil {
|
||||
return err
|
||||
}
|
||||
// Explicit reversed key for "password", independent of vncResponse.
|
||||
cipher, _ := des.NewCipher([]byte{0x0e, 0x86, 0xce, 0xce, 0xee, 0xf6, 0x4e, 0x26})
|
||||
expected := make([]byte, 16)
|
||||
cipher.Encrypt(expected[:8], challenge[:8])
|
||||
cipher.Encrypt(expected[8:], challenge[8:])
|
||||
if !bytes.Equal(response, expected) {
|
||||
return errors.New("incorrect VNC DES response")
|
||||
}
|
||||
}
|
||||
if auth || minor == 8 {
|
||||
if err := stream.write([]byte{0, 0, 0, 0}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := io.ReadFull(stream, answer[:1]); err != nil {
|
||||
return err
|
||||
}
|
||||
if answer[0] != 1 {
|
||||
return errors.New("viewer must use shared mode")
|
||||
}
|
||||
init := make([]byte, 24)
|
||||
binary.BigEndian.PutUint16(init[:2], uint16(width))
|
||||
binary.BigEndian.PutUint16(init[2:4], uint16(height))
|
||||
if err := sendSplit(stream, init); err != nil {
|
||||
return err
|
||||
}
|
||||
format := make([]byte, 20)
|
||||
if _, err := io.ReadFull(stream, format); err != nil {
|
||||
return err
|
||||
}
|
||||
if !bytes.Equal(format, []byte{0, 0, 0, 0, 32, 24, 0, 1, 0, 255, 0, 255, 0, 255, 16, 8, 0, 0, 0, 0}) {
|
||||
return errors.New("pixel format is not little-endian 32bpp truecolor")
|
||||
}
|
||||
encodings := make([]byte, 12)
|
||||
if _, err := io.ReadFull(stream, encodings); err != nil {
|
||||
return err
|
||||
}
|
||||
if !bytes.Equal(encodings, []byte{2, 0, 0, 2, 0, 0, 0, 0, 255, 255, 255, 33}) {
|
||||
return errors.New("unexpected encoding negotiation")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func rawUpdate(x, y, width, height int, color [3]byte) []byte {
|
||||
p := make([]byte, 16+width*height*4)
|
||||
p[3] = 1
|
||||
binary.BigEndian.PutUint16(p[4:6], uint16(x))
|
||||
binary.BigEndian.PutUint16(p[6:8], uint16(y))
|
||||
binary.BigEndian.PutUint16(p[8:10], uint16(width))
|
||||
binary.BigEndian.PutUint16(p[10:12], uint16(height))
|
||||
for i := range width * height {
|
||||
p[16+i*4], p[17+i*4], p[18+i*4] = color[2], color[1], color[0]
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func resizeUpdate(width, height int) []byte {
|
||||
p := rawUpdate(0, 0, 1, 1, [3]byte{})[:16]
|
||||
binary.BigEndian.PutUint16(p[8:10], uint16(width))
|
||||
binary.BigEndian.PutUint16(p[10:12], uint16(height))
|
||||
binary.BigEndian.PutUint32(p[12:16], 0xffffff21)
|
||||
return p
|
||||
}
|
||||
|
||||
func TestRFBVersionsAuthFramesAndGeometry(t *testing.T) {
|
||||
for _, minor := range []int{3, 7, 8} {
|
||||
for _, auth := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("3.%d/auth=%v", minor, auth), func(t *testing.T) {
|
||||
url, result := fixture(t, func(stream *wsStream) error {
|
||||
if err := serverHandshake(stream, minor, 2, 1, auth); err != nil {
|
||||
return err
|
||||
}
|
||||
var request [10]byte
|
||||
if _, err := io.ReadFull(stream, request[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
if request[0] != 3 || request[1] != 0 {
|
||||
return errors.New("initial frame must be a full update")
|
||||
}
|
||||
if err := sendSplit(stream, rawUpdate(0, 0, 1, 1, [3]byte{255, 0, 7})); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := sendSplit(stream, rawUpdate(1, 0, 1, 1, [3]byte{3, 255, 9})); err != nil {
|
||||
return err
|
||||
}
|
||||
return sendSplit(stream, rawUpdate(2, 0, 1, 1, [3]byte{}))
|
||||
})
|
||||
conn, _, err := websocket.DefaultDialer.Dial(url, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
conn.SetReadDeadline(time.Now().Add(5 * time.Second))
|
||||
frame, err := negotiate(&wsStream{conn: conn}, "password")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := frame.request(true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := frame.readMessage(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if frame.complete {
|
||||
t.Fatal("partial framebuffer declared ready")
|
||||
}
|
||||
if err := frame.readMessage(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !frame.complete || !bytes.Equal(frame.pixels, []byte{255, 0, 7, 3, 255, 9}) {
|
||||
t.Fatalf("wrong decoded RGB24 framebuffer: %v", frame.pixels)
|
||||
}
|
||||
if err := frame.readMessage(); err == nil {
|
||||
t.Fatal("out-of-bounds rectangle accepted")
|
||||
}
|
||||
if err := <-result; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRFBRejectsMalformedMessages(t *testing.T) {
|
||||
cases := map[string][]byte{
|
||||
"oversized_desktop": resizeUpdate(maxWidth+1, 1),
|
||||
"zero_desktop": resizeUpdate(0, 1),
|
||||
"rectangle_count": {0, 0, 0xff, 0xff},
|
||||
"oversized_clipboard": {3, 0, 0, 0, 0xff, 0xff, 0xff, 0xff},
|
||||
"unknown_message": {90},
|
||||
}
|
||||
for name, message := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
url, result := fixture(t, func(stream *wsStream) error {
|
||||
if err := serverHandshake(stream, 8, 2, 2, false); err != nil {
|
||||
return err
|
||||
}
|
||||
return stream.write(message)
|
||||
})
|
||||
conn, _, err := websocket.DefaultDialer.Dial(url, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
frame, err := negotiate(&wsStream{conn: conn}, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := frame.readMessage(); err == nil {
|
||||
t.Fatal("malformed RFB message accepted")
|
||||
}
|
||||
if err := <-result; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func requireFFmpeg(t *testing.T) {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("ffmpeg"); err != nil {
|
||||
t.Skip("requires real FFmpeg with libx264 on PATH")
|
||||
}
|
||||
}
|
||||
|
||||
func loadManifest(t *testing.T, directory string) Manifest {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(filepath.Join(directory, "manifest.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(string(data), "password") {
|
||||
t.Fatal("ticket leaked to manifest")
|
||||
}
|
||||
var manifest Manifest
|
||||
if err := json.Unmarshal(data, &manifest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return manifest
|
||||
}
|
||||
|
||||
func TestFFmpegRecordingResizeAndFreshness(t *testing.T) {
|
||||
requireFFmpeg(t)
|
||||
var fullRequests atomic.Int32
|
||||
var resized atomic.Bool
|
||||
url, result := fixture(t, func(stream *wsStream) error {
|
||||
if err := serverHandshake(stream, 8, 16, 16, true); err != nil {
|
||||
return err
|
||||
}
|
||||
width, height := 16, 16
|
||||
for {
|
||||
var request [10]byte
|
||||
if _, err := io.ReadFull(stream, request[:]); err != nil {
|
||||
return nil
|
||||
}
|
||||
if request[0] != 3 {
|
||||
return errors.New("non-viewer input sent")
|
||||
}
|
||||
if request[1] == 0 {
|
||||
fullRequests.Add(1)
|
||||
}
|
||||
if resized.CompareAndSwap(true, false) {
|
||||
width, height = 17, 19
|
||||
if err := stream.write(resizeUpdate(width, height)); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err := stream.write(rawUpdate(0, 0, width, height, [3]byte{255, 0, 0})); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
})
|
||||
directory := t.TempDir()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
session, err := Start(ctx, Config{URL: url, Ticket: "password", Directory: directory, FPS: 5, MaxBytes: 4 << 20})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer session.Close()
|
||||
initial := loadManifest(t, directory)
|
||||
if initial.State != "recording" || initial.StartedAt == nil {
|
||||
t.Fatal("Start returned without durable recording manifest")
|
||||
}
|
||||
first, err := os.ReadFile(filepath.Join(directory, initial.Segments[0].Filename))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !bytes.Contains(first, []byte("moof")) || !bytes.Contains(first, []byte("mdat")) {
|
||||
t.Fatal("Start returned without a real encoded fragment")
|
||||
}
|
||||
resized.Store(true)
|
||||
deadline := time.Now().Add(8 * time.Second)
|
||||
for {
|
||||
manifest := loadManifest(t, directory)
|
||||
if len(manifest.Segments) >= 2 && fullRequests.Load() >= 2 {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("resize or periodic full-frame request missing")
|
||||
}
|
||||
select {
|
||||
case <-session.Done():
|
||||
t.Fatal(session.Err())
|
||||
case <-time.After(100 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
if err := session.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cancel()
|
||||
if err := session.Close(); err != nil {
|
||||
t.Fatalf("completed Close is not idempotent: %v", err)
|
||||
}
|
||||
if err := <-result; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
manifest := loadManifest(t, directory)
|
||||
if manifest.State != "complete" || len(manifest.Segments) != 2 || len(manifest.Gaps) != 1 {
|
||||
t.Fatalf("unexpected final recording: %+v", manifest)
|
||||
}
|
||||
probe, err := exec.LookPath("ffprobe")
|
||||
if err != nil {
|
||||
t.Log("ffprobe absent; encoded fragments validated by recorder")
|
||||
return
|
||||
}
|
||||
for i, segment := range manifest.Segments {
|
||||
path := filepath.Join(directory, segment.Filename)
|
||||
data, err := exec.Command(probe, "-v", "error", "-show_entries", "stream=codec_name,width,height", "-of", "json", path).Output()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var output struct {
|
||||
Streams []struct {
|
||||
Codec string `json:"codec_name"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
} `json:"streams"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &output); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
expectedWidth, expectedHeight := 16, 16
|
||||
if i == 1 {
|
||||
expectedWidth, expectedHeight = 18, 20
|
||||
}
|
||||
if len(output.Streams) != 1 || output.Streams[0].Codec != "h264" || output.Streams[0].Width != expectedWidth || output.Streams[0].Height != expectedHeight {
|
||||
t.Fatalf("wrong encoded stream: %s", data)
|
||||
}
|
||||
pixels, err := exec.Command("ffmpeg", "-v", "error", "-i", path, "-frames:v", "1", "-f", "rawvideo", "-pix_fmt", "rgb24", "pipe:1").Output()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(pixels) != expectedWidth*expectedHeight*3 || pixels[0] < 240 || pixels[1] > 10 || pixels[2] > 10 {
|
||||
t.Fatal("encoded video does not contain the actual red framebuffer")
|
||||
}
|
||||
t.Logf("Real FFmpeg: %s H264 %dx%d frames=%d durable_bytes=%d", segment.Filename, expectedWidth, expectedHeight, segment.Frames, segment.Size)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFFmpegFrozenTransportIsPartial(t *testing.T) {
|
||||
requireFFmpeg(t)
|
||||
url, _ := fixture(t, func(stream *wsStream) error {
|
||||
if err := serverHandshake(stream, 8, 8, 8, false); err != nil {
|
||||
return err
|
||||
}
|
||||
var request [10]byte
|
||||
if _, err := io.ReadFull(stream, request[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := stream.write(rawUpdate(0, 0, 8, 8, [3]byte{0, 255, 0})); err != nil {
|
||||
return err
|
||||
}
|
||||
for {
|
||||
if _, err := io.ReadFull(stream, request[:]); err != nil {
|
||||
return nil
|
||||
}
|
||||
// Transport activity is not proof that the framebuffer is fresh.
|
||||
if err := stream.write([]byte{2}); err != nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
})
|
||||
directory := t.TempDir()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
session, err := Start(ctx, Config{URL: url, Directory: directory, FPS: 2})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
select {
|
||||
case <-session.Done():
|
||||
case <-time.After(13 * time.Second):
|
||||
session.Close()
|
||||
t.Fatal("frozen frame was kept recording indefinitely")
|
||||
}
|
||||
if session.Err() == nil {
|
||||
t.Fatal("frozen recording reported success")
|
||||
}
|
||||
if err := session.Close(); err == nil {
|
||||
t.Fatal("Close hid recording failure")
|
||||
}
|
||||
manifest := loadManifest(t, directory)
|
||||
if manifest.State != "partial" || len(manifest.Gaps) != 1 || manifest.Segments[0].Size == 0 {
|
||||
t.Fatalf("lost recording evidence: %+v", manifest)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStartNeverAcceptsPartialFramebuffer(t *testing.T) {
|
||||
requireFFmpeg(t)
|
||||
url, _ := fixture(t, func(stream *wsStream) error {
|
||||
if err := serverHandshake(stream, 8, 2, 2, false); err != nil {
|
||||
return err
|
||||
}
|
||||
var request [10]byte
|
||||
for {
|
||||
if _, err := io.ReadFull(stream, request[:]); err != nil {
|
||||
return nil
|
||||
}
|
||||
if err := stream.write(rawUpdate(0, 0, 1, 1, [3]byte{255, 0, 0})); err != nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
})
|
||||
directory := t.TempDir()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
|
||||
defer cancel()
|
||||
session, err := Start(ctx, Config{URL: url, Directory: directory})
|
||||
if err == nil {
|
||||
session.Close()
|
||||
t.Fatal("partial framebuffer passed readiness gate")
|
||||
}
|
||||
manifest := loadManifest(t, directory)
|
||||
if manifest.State != "unavailable" || len(manifest.Segments) != 0 {
|
||||
t.Fatalf("partial framebuffer created misleading video: %+v", manifest)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFFmpegByteLimitRetainsEvidence(t *testing.T) {
|
||||
requireFFmpeg(t)
|
||||
url, _ := fixture(t, func(stream *wsStream) error {
|
||||
if err := serverHandshake(stream, 8, 256, 256, false); err != nil {
|
||||
return err
|
||||
}
|
||||
var request [10]byte
|
||||
seed := uint32(1)
|
||||
for {
|
||||
if _, err := io.ReadFull(stream, request[:]); err != nil {
|
||||
return nil
|
||||
}
|
||||
frame := rawUpdate(0, 0, 256, 256, [3]byte{})
|
||||
for i := 16; i < len(frame); i++ {
|
||||
seed ^= seed << 13
|
||||
seed ^= seed >> 17
|
||||
seed ^= seed << 5
|
||||
frame[i] = byte(seed)
|
||||
}
|
||||
if err := stream.write(frame); err != nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
})
|
||||
directory := t.TempDir()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
session, err := Start(ctx, Config{URL: url, Directory: directory, FPS: 5, MaxBytes: 1 << 20})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer session.Close()
|
||||
select {
|
||||
case <-session.Done():
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("recording exceeded byte budget without stopping")
|
||||
}
|
||||
if session.Err() == nil {
|
||||
t.Fatal("byte-limited recording reported success")
|
||||
}
|
||||
manifest := loadManifest(t, directory)
|
||||
if manifest.State != "partial" || manifest.Segments[0].Size == 0 {
|
||||
t.Fatalf("byte limit discarded partial evidence: %+v", manifest)
|
||||
}
|
||||
entries, err := os.ReadDir(directory)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var total int64
|
||||
for _, entry := range entries {
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
total += info.Size()
|
||||
}
|
||||
if total > 1<<20 {
|
||||
t.Fatalf("private recording exceeded limit: %d bytes", total)
|
||||
}
|
||||
t.Logf("Byte limit stopped real encoder; retained %d bytes of partial evidence", total)
|
||||
}
|
||||
|
||||
func TestFFmpegCancellationCannotBecomeComplete(t *testing.T) {
|
||||
requireFFmpeg(t)
|
||||
url, _ := fixture(t, func(stream *wsStream) error {
|
||||
if err := serverHandshake(stream, 8, 8, 8, false); err != nil {
|
||||
return err
|
||||
}
|
||||
var request [10]byte
|
||||
for {
|
||||
if _, err := io.ReadFull(stream, request[:]); err != nil {
|
||||
return nil
|
||||
}
|
||||
if err := stream.write(rawUpdate(0, 0, 8, 8, [3]byte{0, 0, 255})); err != nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
})
|
||||
directory := t.TempDir()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
session, err := Start(ctx, Config{URL: url, Directory: directory})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cancel()
|
||||
if err := session.Close(); err == nil {
|
||||
t.Fatal("Close hid external cancellation")
|
||||
}
|
||||
if manifest := loadManifest(t, directory); manifest.State != "partial" {
|
||||
t.Fatalf("cancelled recording marked %q", manifest.State)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
package recorder
|
||||
|
||||
import (
|
||||
"crypto/des"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"io"
|
||||
"math/bits"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/gorilla/websocket"
|
||||
)
|
||||
|
||||
const (
|
||||
maxWidth = 4096
|
||||
maxHeight = 2160
|
||||
maxPixels = maxWidth * maxHeight
|
||||
maxMessageBytes = 64 << 20
|
||||
freshnessLimit = 10 * time.Second
|
||||
fullUpdateInterval = 2 * time.Second
|
||||
)
|
||||
|
||||
// wsStream joins binary WebSocket messages without treating message boundaries as
|
||||
// RFB boundaries. Neither authentication errors nor peer-supplied text is logged.
|
||||
type wsStream struct {
|
||||
conn *websocket.Conn
|
||||
reader io.Reader
|
||||
}
|
||||
|
||||
func (s *wsStream) Read(p []byte) (int, error) {
|
||||
for {
|
||||
if s.reader == nil {
|
||||
kind, reader, err := s.conn.NextReader()
|
||||
if err != nil {
|
||||
return 0, errors.New("RFB transport closed or timed out")
|
||||
}
|
||||
if kind != websocket.BinaryMessage {
|
||||
return 0, errors.New("RFB requires binary WebSocket messages")
|
||||
}
|
||||
s.reader = reader
|
||||
}
|
||||
n, err := s.reader.Read(p)
|
||||
if err == io.EOF {
|
||||
s.reader = nil
|
||||
if n > 0 {
|
||||
return n, nil
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return n, errors.New("RFB transport read failed")
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (s *wsStream) write(p []byte) error {
|
||||
if err := s.conn.SetWriteDeadline(time.Now().Add(5 * time.Second)); err != nil {
|
||||
return errors.New("RFB write deadline failed")
|
||||
}
|
||||
if err := s.conn.WriteMessage(websocket.BinaryMessage, p); err != nil {
|
||||
return errors.New("RFB transport write failed")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type framebuffer struct {
|
||||
stream *wsStream
|
||||
mu sync.Mutex
|
||||
width, height int
|
||||
pixels []byte
|
||||
coverage []uint64
|
||||
covered int
|
||||
complete bool
|
||||
awaitingFull bool
|
||||
generation uint64
|
||||
updates atomic.Uint64
|
||||
lastFull atomic.Int64
|
||||
}
|
||||
|
||||
func validGeometry(width, height int) bool {
|
||||
return width > 0 && height > 0 && width <= maxWidth && height <= maxHeight && width*height <= maxPixels
|
||||
}
|
||||
|
||||
func (f *framebuffer) resize(width, height int) error {
|
||||
if !validGeometry(width, height) {
|
||||
return errors.New("RFB framebuffer dimensions exceed limits")
|
||||
}
|
||||
f.width, f.height = width, height
|
||||
f.pixels = make([]byte, width*height*3)
|
||||
f.coverage = make([]uint64, (width*height+63)/64)
|
||||
f.covered = 0
|
||||
f.complete = false
|
||||
f.awaitingFull = true
|
||||
f.generation++
|
||||
return nil
|
||||
}
|
||||
|
||||
func vncResponse(ticket string, challenge []byte) []byte {
|
||||
var key [8]byte
|
||||
copy(key[:], ticket)
|
||||
for i := range key {
|
||||
key[i] = bits.Reverse8(key[i])
|
||||
}
|
||||
cipher, _ := des.NewCipher(key[:]) // DES always accepts an eight-byte key.
|
||||
response := make([]byte, 16)
|
||||
cipher.Encrypt(response[:8], challenge[:8])
|
||||
cipher.Encrypt(response[8:], challenge[8:])
|
||||
return response
|
||||
}
|
||||
|
||||
func negotiate(stream *wsStream, ticket string) (*framebuffer, error) {
|
||||
var version [12]byte
|
||||
if _, err := io.ReadFull(stream, version[:]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
minor := 0
|
||||
switch string(version[:]) {
|
||||
case "RFB 003.003\n":
|
||||
minor = 3
|
||||
case "RFB 003.007\n":
|
||||
minor = 7
|
||||
case "RFB 003.008\n":
|
||||
minor = 8
|
||||
default:
|
||||
return nil, errors.New("unsupported RFB protocol version")
|
||||
}
|
||||
if err := stream.write(version[:]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
security := byte(0)
|
||||
if minor == 3 {
|
||||
var offer [4]byte
|
||||
if _, err := io.ReadFull(stream, offer[:]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
typ := binary.BigEndian.Uint32(offer[:])
|
||||
if typ != 1 && typ != 2 {
|
||||
return nil, errors.New("unsupported or rejected RFB security")
|
||||
}
|
||||
security = byte(typ)
|
||||
} else {
|
||||
var count [1]byte
|
||||
if _, err := io.ReadFull(stream, count[:]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if count[0] == 0 {
|
||||
return nil, errors.New("RFB authentication rejected")
|
||||
}
|
||||
offers := make([]byte, int(count[0]))
|
||||
if _, err := io.ReadFull(stream, offers); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, typ := range offers {
|
||||
if typ == 2 && ticket != "" {
|
||||
security = 2
|
||||
break
|
||||
}
|
||||
if typ == 1 {
|
||||
security = 1
|
||||
}
|
||||
}
|
||||
if security == 0 {
|
||||
return nil, errors.New("no supported RFB authentication method")
|
||||
}
|
||||
if err := stream.write([]byte{security}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if security == 2 {
|
||||
if ticket == "" {
|
||||
return nil, errors.New("RFB authentication ticket required")
|
||||
}
|
||||
var challenge [16]byte
|
||||
if _, err := io.ReadFull(stream, challenge[:]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := stream.write(vncResponse(ticket, challenge[:])); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if security == 2 || minor == 8 {
|
||||
var result [4]byte
|
||||
if _, err := io.ReadFull(stream, result[:]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if binary.BigEndian.Uint32(result[:]) != 0 {
|
||||
return nil, errors.New("RFB authentication rejected")
|
||||
}
|
||||
}
|
||||
if err := stream.write([]byte{1}); err != nil {
|
||||
return nil, err
|
||||
} // Shared; never evict an existing viewer.
|
||||
var init [24]byte
|
||||
if _, err := io.ReadFull(stream, init[:]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
nameLength := binary.BigEndian.Uint32(init[20:24])
|
||||
if nameLength > 65536 {
|
||||
return nil, errors.New("RFB desktop name exceeds limit")
|
||||
}
|
||||
if _, err := io.CopyN(io.Discard, stream, int64(nameLength)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
f := &framebuffer{stream: stream}
|
||||
if err := f.resize(int(binary.BigEndian.Uint16(init[:2])), int(binary.BigEndian.Uint16(init[2:4]))); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// 32bpp, depth 24, little endian, truecolor; RGB maxima 255 and
|
||||
// shifts 16/8/0 mean the wire bytes are B,G,R,padding.
|
||||
pixelFormat := []byte{0, 0, 0, 0, 32, 24, 0, 1, 0, 255, 0, 255, 0, 255, 16, 8, 0, 0, 0, 0}
|
||||
if err := stream.write(pixelFormat); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
encodings := []byte{2, 0, 0, 2, 0, 0, 0, 0, 255, 255, 255, 33} // RAW, DesktopSize (-223).
|
||||
if err := stream.write(encodings); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
f.lastFull.Store(time.Now().UnixNano())
|
||||
return f, nil
|
||||
}
|
||||
|
||||
func (f *framebuffer) request(full bool) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if full && !f.awaitingFull {
|
||||
clear(f.coverage)
|
||||
f.covered = 0
|
||||
f.awaitingFull = true
|
||||
}
|
||||
var request [10]byte
|
||||
request[0] = 3
|
||||
if !full {
|
||||
request[1] = 1
|
||||
}
|
||||
binary.BigEndian.PutUint16(request[6:8], uint16(f.width))
|
||||
binary.BigEndian.PutUint16(request[8:10], uint16(f.height))
|
||||
return f.stream.write(request[:])
|
||||
}
|
||||
|
||||
func (f *framebuffer) readMessage() error {
|
||||
var typ [1]byte
|
||||
if _, err := io.ReadFull(f.stream, typ[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
switch typ[0] {
|
||||
case 0:
|
||||
return f.readUpdate()
|
||||
case 2: // Bell carries no data and never counts as a fresh frame.
|
||||
return nil
|
||||
case 3: // Clipboard is not collected, displayed, or sent back.
|
||||
var header [7]byte
|
||||
if _, err := io.ReadFull(f.stream, header[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
n := binary.BigEndian.Uint32(header[3:])
|
||||
if n > 1<<20 {
|
||||
return errors.New("RFB clipboard exceeds limit")
|
||||
}
|
||||
_, err := io.CopyN(io.Discard, f.stream, int64(n))
|
||||
return err
|
||||
default:
|
||||
return errors.New("unsupported RFB server message")
|
||||
}
|
||||
}
|
||||
|
||||
func (f *framebuffer) readUpdate() error {
|
||||
var header [3]byte
|
||||
if _, err := io.ReadFull(f.stream, header[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
count := int(binary.BigEndian.Uint16(header[1:]))
|
||||
if count > 4096 {
|
||||
return errors.New("RFB rectangle count exceeds limit")
|
||||
}
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
var row [maxWidth * 4]byte
|
||||
var rectangle [12]byte
|
||||
var total int64
|
||||
for i := range count {
|
||||
if _, err := io.ReadFull(f.stream, rectangle[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
x, y := int(binary.BigEndian.Uint16(rectangle[:2])), int(binary.BigEndian.Uint16(rectangle[2:4]))
|
||||
w, h := int(binary.BigEndian.Uint16(rectangle[4:6])), int(binary.BigEndian.Uint16(rectangle[6:8]))
|
||||
encoding := int32(binary.BigEndian.Uint32(rectangle[8:12]))
|
||||
if encoding == -223 {
|
||||
if x != 0 || y != 0 || i != count-1 {
|
||||
return errors.New("invalid RFB DesktopSize rectangle")
|
||||
}
|
||||
if err := f.resize(w, h); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
if encoding != 0 {
|
||||
return errors.New("unrequested RFB rectangle encoding")
|
||||
}
|
||||
if w <= 0 || h <= 0 || x+w > f.width || y+h > f.height {
|
||||
return errors.New("RFB rectangle outside framebuffer")
|
||||
}
|
||||
total += int64(w) * int64(h) * 4
|
||||
if total > maxMessageBytes {
|
||||
return errors.New("RFB update exceeds limit")
|
||||
}
|
||||
for dy := range h {
|
||||
if _, err := io.ReadFull(f.stream, row[:w*4]); err != nil {
|
||||
return err
|
||||
}
|
||||
start := (y+dy)*f.width + x
|
||||
for dx := range w {
|
||||
pixel := start + dx
|
||||
offset := pixel * 3
|
||||
f.pixels[offset], f.pixels[offset+1], f.pixels[offset+2] = row[dx*4+2], row[dx*4+1], row[dx*4]
|
||||
if f.awaitingFull {
|
||||
mask := uint64(1) << (pixel & 63)
|
||||
if f.coverage[pixel/64]&mask == 0 {
|
||||
f.coverage[pixel/64] |= mask
|
||||
f.covered++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if f.awaitingFull && f.covered == f.width*f.height {
|
||||
f.complete = true
|
||||
f.awaitingFull = false
|
||||
f.lastFull.Store(time.Now().UnixNano())
|
||||
}
|
||||
f.updates.Add(1)
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
CREATE TABLE IF NOT EXISTS users(id uuid PRIMARY KEY, username text UNIQUE NOT NULL, password_hash text NOT NULL, role text NOT NULL CHECK(role IN ('admin','operator')), disabled boolean NOT NULL DEFAULT false, created_at timestamptz NOT NULL DEFAULT now());
|
||||
CREATE TABLE IF NOT EXISTS sessions(token_hash text PRIMARY KEY, user_id uuid NOT NULL REFERENCES users(id), csrf_token text NOT NULL, expires_at timestamptz NOT NULL);
|
||||
CREATE TABLE IF NOT EXISTS uploads(id uuid PRIMARY KEY, owner_id uuid NOT NULL REFERENCES users(id), filename text NOT NULL, size bigint NOT NULL CHECK(size > 0), sha256 text NOT NULL, storage_key text NOT NULL UNIQUE, created_at timestamptz NOT NULL DEFAULT now());
|
||||
CREATE TABLE IF NOT EXISTS profiles(id uuid PRIMARY KEY, name text NOT NULL, os text NOT NULL, architecture text NOT NULL CHECK(architecture IN ('x64','x86')), enabled boolean NOT NULL DEFAULT false, qualification text NOT NULL DEFAULT 'unqualified', state text NOT NULL DEFAULT 'maintenance', current_revision_id uuid, online_available boolean NOT NULL DEFAULT false, reason text NOT NULL DEFAULT 'Source setup and qualification required');
|
||||
CREATE TABLE IF NOT EXISTS revisions(id uuid PRIMARY KEY, profile_id uuid NOT NULL REFERENCES profiles(id), source_ref text NOT NULL, fingerprint text NOT NULL, config_digest text NOT NULL, qualification jsonb, created_at timestamptz NOT NULL DEFAULT now());
|
||||
CREATE TABLE IF NOT EXISTS bindings(owner_id uuid PRIMARY KEY REFERENCES users(id), pool text NOT NULL, iso_storage text NOT NULL, disk_storage text NOT NULL, node text NOT NULL, configured boolean NOT NULL DEFAULT false, reason text NOT NULL DEFAULT 'Worker credentials not validated');
|
||||
CREATE TABLE IF NOT EXISTS jobs(id uuid PRIMARY KEY, owner_id uuid NOT NULL REFERENCES users(id), upload_id uuid NOT NULL REFERENCES uploads(id), execution_filename text NOT NULL, settings jsonb NOT NULL, status text NOT NULL DEFAULT 'queued', cancel_requested boolean NOT NULL DEFAULT false, created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now());
|
||||
CREATE TABLE IF NOT EXISTS runs(id uuid PRIMARY KEY, job_id uuid NOT NULL REFERENCES jobs(id), profile_id uuid NOT NULL REFERENCES profiles(id), revision_id uuid NOT NULL REFERENCES revisions(id), antivirus text NOT NULL DEFAULT 'defender', status text NOT NULL DEFAULT 'queued');
|
||||
CREATE TABLE IF NOT EXISTS attempts(id uuid PRIMARY KEY, run_id uuid NOT NULL REFERENCES runs(id), command_id uuid NOT NULL UNIQUE, phase text NOT NULL DEFAULT 'queued', outcome text NOT NULL DEFAULT 'pending', findings text NOT NULL DEFAULT 'unknown', telemetry text NOT NULL DEFAULT 'pending', cleanup text NOT NULL DEFAULT 'pending', error text NOT NULL DEFAULT '', created_at timestamptz NOT NULL DEFAULT now(), started_at timestamptz, finished_at timestamptz, deadline_at timestamptz, lease_owner text, lease_until timestamptz, release_requested boolean NOT NULL DEFAULT false, video jsonb NOT NULL DEFAULT '{"state":"pending","segments":[],"gaps":[],"started_at":null,"finished_at":null}');
|
||||
CREATE TABLE IF NOT EXISTS allocations(id uuid PRIMARY KEY, attempt_id uuid NOT NULL UNIQUE REFERENCES attempts(id), owner_id uuid NOT NULL REFERENCES users(id), node text NOT NULL, vmid integer NOT NULL CHECK(vmid NOT IN (7000,7001)), pool text NOT NULL, state text NOT NULL, upid text, metadata jsonb NOT NULL DEFAULT '{}');
|
||||
CREATE TABLE IF NOT EXISTS media(id uuid PRIMARY KEY, job_id uuid NOT NULL REFERENCES jobs(id), owner_id uuid NOT NULL REFERENCES users(id), node text NOT NULL, volume text NOT NULL, state text NOT NULL, UNIQUE(job_id,node));
|
||||
CREATE TABLE IF NOT EXISTS artifacts(id uuid PRIMARY KEY, job_id uuid NOT NULL REFERENCES jobs(id), attempt_id uuid REFERENCES attempts(id), kind text NOT NULL, filename text NOT NULL, content_type text NOT NULL, size bigint NOT NULL, sha256 text NOT NULL, storage_key text NOT NULL UNIQUE, created_at timestamptz NOT NULL DEFAULT now());
|
||||
CREATE TABLE IF NOT EXISTS events(id bigserial PRIMARY KEY, job_id uuid REFERENCES jobs(id), attempt_id uuid REFERENCES attempts(id), kind text NOT NULL, message text NOT NULL, created_at timestamptz NOT NULL DEFAULT now());
|
||||
CREATE TABLE IF NOT EXISTS qualifications(id uuid PRIMARY KEY, profile_id uuid NOT NULL REFERENCES profiles(id), status text NOT NULL DEFAULT 'queued', result jsonb, created_at timestamptz NOT NULL DEFAULT now());
|
||||
CREATE TABLE IF NOT EXISTS heartbeats(name text PRIMARY KEY, seen_at timestamptz NOT NULL, ready boolean NOT NULL, blockers jsonb NOT NULL DEFAULT '[]');
|
||||
CREATE TABLE IF NOT EXISTS login_limits(key text PRIMARY KEY, failures integer NOT NULL DEFAULT 0, reset_at timestamptz NOT NULL);
|
||||
CREATE INDEX IF NOT EXISTS attempts_lease ON attempts(phase,lease_until);
|
||||
CREATE INDEX IF NOT EXISTS jobs_owner ON jobs(owner_id,created_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS artifacts_job ON artifacts(job_id);
|
||||
CREATE INDEX IF NOT EXISTS events_job ON events(job_id,id);
|
||||
ALTER TABLE attempts ADD COLUMN IF NOT EXISTS report jsonb;
|
||||
ALTER TABLE profiles ADD COLUMN IF NOT EXISTS metadata jsonb NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE allocations DROP CONSTRAINT IF EXISTS allocations_vmid_key;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS allocations_active_vmid ON allocations(vmid) WHERE state <> 'deleted';
|
||||
CREATE TABLE IF NOT EXISTS extractor_allocations(id uuid PRIMARY KEY, attempt_id uuid UNIQUE NOT NULL REFERENCES attempts(id), owner_id uuid NOT NULL REFERENCES users(id), node text NOT NULL, vmid integer NOT NULL CHECK(vmid NOT IN(7000,7001)), state text NOT NULL, upid text, disk_slot text NOT NULL, disk_serial text NOT NULL, disk_bytes bigint NOT NULL, metadata jsonb NOT NULL DEFAULT '{}');
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS extractor_active_vmid ON extractor_allocations(vmid) WHERE state <> 'deleted';
|
||||
ALTER TABLE bindings ADD COLUMN IF NOT EXISTS isolation_expires_at timestamptz;
|
||||
@@ -0,0 +1,74 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"embed"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
//go:embed schema.sql
|
||||
var schema embed.FS
|
||||
|
||||
func Open(ctx context.Context, url string) (*pgxpool.Pool, error) {
|
||||
p, e := pgxpool.New(ctx, url)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if e = p.Ping(ctx); e != nil {
|
||||
p.Close()
|
||||
return nil, e
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
func Migrate(ctx context.Context, p *pgxpool.Pool) error {
|
||||
b, e := schema.ReadFile("schema.sql")
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
tx, e := p.Begin(ctx)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
if _, e = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(74638291)"); e != nil {
|
||||
return e
|
||||
}
|
||||
if _, e = tx.Exec(ctx, string(b)); e != nil {
|
||||
return e
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
func NewID() string {
|
||||
var b [16]byte
|
||||
if _, e := rand.Read(b[:]); e != nil {
|
||||
panic(e)
|
||||
}
|
||||
b[6] = (b[6] & 15) | 64
|
||||
b[8] = (b[8] & 63) | 128
|
||||
h := hex.EncodeToString(b[:])
|
||||
return h[:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:]
|
||||
}
|
||||
func Secret(name string) (string, error) {
|
||||
if f := os.Getenv(name + "_FILE"); f != "" {
|
||||
b, e := os.ReadFile(f)
|
||||
if e != nil {
|
||||
return "", e
|
||||
}
|
||||
v := strings.TrimSpace(string(b))
|
||||
if v == "" {
|
||||
return "", fmt.Errorf("empty secret file for %s", name)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
v := os.Getenv(name)
|
||||
if v == "" {
|
||||
return "", fmt.Errorf("%s or %s_FILE required", name, name)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func (e *engine) publishFile(ctx context.Context, a attempt, kind, filename, ctype, path string) error {
|
||||
info, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return errors.New("artifact must be a regular private file")
|
||||
}
|
||||
key, err := filepath.Rel(e.root, path)
|
||||
if err != nil || strings.HasPrefix(key, "..") || filepath.IsAbs(key) {
|
||||
return errors.New("artifact escaped private root")
|
||||
}
|
||||
if len(kind) > 64 || len(filename) > 240 || len(ctype) > 200 {
|
||||
return errors.New("artifact metadata exceeds bounds")
|
||||
}
|
||||
if ctype == "" {
|
||||
ctype = "application/octet-stream"
|
||||
}
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(h, f)
|
||||
f.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
hash := hex.EncodeToString(h.Sum(nil))
|
||||
key = filepath.ToSlash(key)
|
||||
var existing string
|
||||
err = e.db.QueryRow(ctx, "SELECT sha256 FROM artifacts WHERE storage_key=$1", key).Scan(&existing)
|
||||
if err == nil {
|
||||
if existing != hash {
|
||||
return errors.New("published artifact was modified")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if !isNoRows(err) {
|
||||
return err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "INSERT INTO artifacts(id,job_id,attempt_id,kind,filename,content_type,size,sha256,storage_key) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9)", newID(), a.JobID, a.ID, kind, filename, ctype, n, hash, key)
|
||||
return err
|
||||
}
|
||||
func (e *engine) publishVideo(ctx context.Context, a attempt, dir string, complete bool) error {
|
||||
raw, err := os.ReadFile(filepath.Join(dir, "manifest.json"))
|
||||
if os.IsNotExist(err) {
|
||||
_, err = e.db.Exec(ctx, "UPDATE attempts SET video=jsonb_build_object('state','unavailable','segments','[]'::jsonb,'gaps','[]'::jsonb,'started_at',null,'finished_at',now()) WHERE id=$1", a.ID)
|
||||
return err
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(raw) > 1<<20 {
|
||||
return errors.New("recorder manifest exceeds bound")
|
||||
}
|
||||
var manifest struct {
|
||||
State string `json:"state"`
|
||||
StartedAt any `json:"started_at"`
|
||||
FinishedAt any `json:"finished_at"`
|
||||
Gaps []map[string]any `json:"gaps"`
|
||||
Segments []struct {
|
||||
Filename string `json:"filename"`
|
||||
} `json:"segments"`
|
||||
}
|
||||
if json.Unmarshal(raw, &manifest) != nil {
|
||||
return errors.New("invalid recorder manifest")
|
||||
}
|
||||
if len(manifest.Segments) > 2000 {
|
||||
return errors.New("recorder segment count exceeded")
|
||||
}
|
||||
for _, s := range manifest.Segments {
|
||||
if filepath.Base(s.Filename) != s.Filename || !strings.HasPrefix(s.Filename, "segment-") || !strings.HasSuffix(s.Filename, ".mp4") {
|
||||
return errors.New("unsafe recorder segment name")
|
||||
}
|
||||
if err = e.publishFile(ctx, a, "video", s.Filename, "video/mp4", filepath.Join(dir, s.Filename)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err = e.publishFile(ctx, a, "video_manifest", "video-manifest.json", "application/json", filepath.Join(dir, "manifest.json")); err != nil {
|
||||
return err
|
||||
}
|
||||
rows, err := e.db.Query(ctx, "SELECT id,filename,content_type,size,sha256,created_at FROM artifacts WHERE attempt_id=$1 AND kind='video' ORDER BY filename", a.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer rows.Close()
|
||||
segments := []map[string]any{}
|
||||
for rows.Next() {
|
||||
var id, name, ct, sha string
|
||||
var size int64
|
||||
var created any
|
||||
if err = rows.Scan(&id, &name, &ct, &size, &sha, &created); err != nil {
|
||||
return err
|
||||
}
|
||||
segments = append(segments, map[string]any{"id": id, "job_id": a.JobID, "attempt_id": a.ID, "kind": "video", "filename": name, "content_type": ct, "size": size, "sha256": sha, "created_at": created, "url": "/api/v1/artifacts/" + id + "/content"})
|
||||
}
|
||||
if err = rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
state := manifest.State
|
||||
if !complete || state != "complete" {
|
||||
state = "partial"
|
||||
if len(segments) == 0 {
|
||||
state = "unavailable"
|
||||
}
|
||||
}
|
||||
if manifest.Gaps == nil {
|
||||
manifest.Gaps = []map[string]any{}
|
||||
}
|
||||
if !complete {
|
||||
manifest.Gaps = append(manifest.Gaps, map[string]any{"at": manifest.FinishedAt, "reason": "Attempt observation ended abnormally; no continuity claimed"})
|
||||
}
|
||||
video, _ := json.Marshal(map[string]any{"state": state, "segments": segments, "gaps": manifest.Gaps, "started_at": manifest.StartedAt, "finished_at": manifest.FinishedAt})
|
||||
_, err = e.db.Exec(ctx, "UPDATE attempts SET video=$2 WHERE id=$1", a.ID, video)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if complete && state != "complete" {
|
||||
return errors.New("recorder output is incomplete")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/url"
|
||||
"otche/internal/pve"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Adoption is permitted only for our durable clone intent plus terminal OK UPID,
|
||||
// exact requested name and owner pool. Never retroactively tags arbitrary VMIDs.
|
||||
func (e *engine) adoptCompletedClone(ctx context.Context, a attempt, l allocation, b Binding, cs clients) error {
|
||||
if l.UPID == nil {
|
||||
return errors.New("clone task acceptance unknown; automatic adoption prohibited")
|
||||
}
|
||||
if err := cs.provisioner.Task(ctx, b.Node, *l.UPID); err != nil {
|
||||
return err
|
||||
}
|
||||
var meta struct {
|
||||
SourceConfig map[string]any `json:"source_config"`
|
||||
}
|
||||
if json.Unmarshal(l.Metadata, &meta) != nil || meta.SourceConfig == nil {
|
||||
return errors.New("clone source configuration record missing")
|
||||
}
|
||||
state, err := cs.provisioner.Status(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if state != "stopped" {
|
||||
return errors.New("new clone adoption requires stopped VM")
|
||||
}
|
||||
o := e.own(a, l, b)
|
||||
cfg, err := cs.provisioner.Config(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if pve.Text(cfg["name"]) != o.Name {
|
||||
return errors.New("clone name ownership mismatch")
|
||||
}
|
||||
var pool struct {
|
||||
Members []struct {
|
||||
VMID int `json:"vmid"`
|
||||
Node string `json:"node"`
|
||||
Type string `json:"type"`
|
||||
} `json:"members"`
|
||||
}
|
||||
if err = cs.provisioner.Do(ctx, "GET", "/pools/"+b.Pool, nil, &pool); err != nil {
|
||||
return err
|
||||
}
|
||||
member := false
|
||||
for _, m := range pool.Members {
|
||||
if m.VMID == l.VMID && m.Node == b.Node && m.Type == "qemu" {
|
||||
member = true
|
||||
}
|
||||
}
|
||||
if !member {
|
||||
return errors.New("new clone is outside owner pool")
|
||||
}
|
||||
for key, value := range cfg {
|
||||
if key == "scsihw" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(key, "scsi") || strings.HasPrefix(key, "virtio") || strings.HasPrefix(key, "efidisk") || strings.HasPrefix(key, "tpmstate") || strings.HasPrefix(key, "sata") || strings.HasPrefix(key, "ide") {
|
||||
v := pve.Text(value)
|
||||
if strings.Contains(v, "media=cdrom") {
|
||||
continue
|
||||
}
|
||||
if !ownVolume(v, b.DiskStorage, l.VMID) || pve.Text(meta.SourceConfig[key]) == v {
|
||||
return errors.New("clone disk independence/storage/volume-owner verification failed")
|
||||
}
|
||||
}
|
||||
}
|
||||
if err = e.leaseAlive(ctx, a); err != nil {
|
||||
return err
|
||||
}
|
||||
var upid string
|
||||
if err = cs.provisioner.Do(ctx, "POST", pve.VMPath(b.Node, l.VMID)+"/config", url.Values{"tags": {o.Tags()}, "onboot": {"0"}}, &upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.CheckOwned(ctx, o); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE allocations SET state='cloned' WHERE id=$1 AND EXISTS(SELECT 1 FROM attempts WHERE id=$2 AND lease_owner=$3 AND lease_until>now())", l.ID, a.ID, e.id)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,250 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"otche/internal/pve"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Sources map[string]Source `json:"sources"`
|
||||
Owners map[string]Binding `json:"owners"`
|
||||
Xorriso string `json:"xorriso"`
|
||||
Concurrent int `json:"concurrent"`
|
||||
MinArtifactFreeBytes int64 `json:"min_artifact_free_bytes"`
|
||||
MaxArtifactBytes int64 `json:"max_artifact_bytes"`
|
||||
MaxVideoBytes int64 `json:"max_video_bytes"`
|
||||
TaskTimeoutSeconds int `json:"task_timeout_seconds"`
|
||||
PrepareTimeoutSeconds int `json:"prepare_timeout_seconds"`
|
||||
CollectTimeoutSeconds int `json:"collect_timeout_seconds"`
|
||||
WatchdogRequired bool `json:"watchdog_required"`
|
||||
QualificationOwner string `json:"qualification_owner"`
|
||||
BenignFixture string `json:"benign_fixture"`
|
||||
BenignSHA256 string `json:"benign_sha256"`
|
||||
Extractor *ExtractorConfig `json:"extractor"`
|
||||
}
|
||||
type Source struct {
|
||||
Node string `json:"node"`
|
||||
VMID int `json:"vmid"`
|
||||
OwnerID string `json:"owner_id"`
|
||||
SealPath string `json:"seal_path"`
|
||||
CDSlot string `json:"cd_slot"`
|
||||
MaxDiskBytes int64 `json:"max_disk_bytes"`
|
||||
AllowedBridges []string `json:"allowed_bridges"`
|
||||
}
|
||||
type Binding struct {
|
||||
Endpoint string `json:"endpoint"`
|
||||
CAFile string `json:"ca_file"`
|
||||
Node string `json:"node"`
|
||||
Pool string `json:"pool"`
|
||||
ISOStorage string `json:"iso_storage"`
|
||||
DiskStorage string `json:"disk_storage"`
|
||||
ProvisionerFile string `json:"provisioner_file"`
|
||||
RuntimeFile string `json:"runtime_file"`
|
||||
RecorderFile string `json:"recorder_file"`
|
||||
UploaderFile string `json:"uploader_file"`
|
||||
HousekeepingFile string `json:"housekeeping_file"`
|
||||
MinStorageFreeBytes int64 `json:"min_storage_free_bytes"`
|
||||
MinMemoryFreeBytes int64 `json:"min_memory_free_bytes"`
|
||||
MaxOwnedDiskBytes int64 `json:"max_owned_disk_bytes"`
|
||||
IsolationProofFile string `json:"isolation_proof_file"`
|
||||
Online *OnlinePolicy `json:"online"`
|
||||
MaxOwnedArtifactBytes int64 `json:"max_owned_artifact_bytes"`
|
||||
}
|
||||
|
||||
// IsolationProof is generated by operator-run negative ACL/network probes, not by
|
||||
// admission. It is bound to the exact endpoint, owner, node and storage identities.
|
||||
type IsolationProof struct {
|
||||
OwnerID string `json:"owner_id"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
Node string `json:"node"`
|
||||
Pool string `json:"pool"`
|
||||
ISOStorage string `json:"iso_storage"`
|
||||
DiskStorage string `json:"disk_storage"`
|
||||
Passed bool `json:"passed"`
|
||||
ExpiresAt time.Time `json:"expires_at"`
|
||||
EvidenceSHA256 string `json:"evidence_sha256"`
|
||||
}
|
||||
|
||||
var ident = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
|
||||
var uuidRE = regexp.MustCompile(`^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$`)
|
||||
var shaRE = regexp.MustCompile(`^[a-f0-9]{64}$`)
|
||||
|
||||
func newID() string {
|
||||
var b [16]byte
|
||||
if _, err := rand.Read(b[:]); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
b[6] = (b[6] & 15) | 64
|
||||
b[8] = (b[8] & 63) | 128
|
||||
h := hex.EncodeToString(b[:])
|
||||
return h[:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:]
|
||||
}
|
||||
func loadConfig(path string) (Config, error) {
|
||||
var c Config
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return c, errors.New("worker configuration file unavailable")
|
||||
}
|
||||
if len(b) > 1<<20 {
|
||||
return c, errors.New("worker configuration exceeds bound")
|
||||
}
|
||||
if err = json.Unmarshal(b, &c); err != nil {
|
||||
return c, errors.New("worker configuration invalid")
|
||||
}
|
||||
if c.Concurrent == 0 {
|
||||
c.Concurrent = 1
|
||||
}
|
||||
if c.Concurrent < 1 || c.Concurrent > 16 {
|
||||
return c, errors.New("invalid concurrency limit")
|
||||
}
|
||||
if c.Xorriso == "" {
|
||||
c.Xorriso = "xorriso"
|
||||
}
|
||||
if c.TaskTimeoutSeconds == 0 {
|
||||
c.TaskTimeoutSeconds = 1200
|
||||
}
|
||||
if c.PrepareTimeoutSeconds == 0 {
|
||||
c.PrepareTimeoutSeconds = 180
|
||||
}
|
||||
if c.CollectTimeoutSeconds == 0 {
|
||||
c.CollectTimeoutSeconds = 60
|
||||
}
|
||||
if c.MaxArtifactBytes == 0 {
|
||||
c.MaxArtifactBytes = 64 << 20
|
||||
}
|
||||
if c.MaxVideoBytes == 0 {
|
||||
c.MaxVideoBytes = 2 << 30
|
||||
}
|
||||
if c.MinArtifactFreeBytes == 0 {
|
||||
c.MinArtifactFreeBytes = 4 << 30
|
||||
}
|
||||
if c.TaskTimeoutSeconds < 30 || c.TaskTimeoutSeconds > 3600 || c.PrepareTimeoutSeconds < 30 || c.PrepareTimeoutSeconds > 900 || c.CollectTimeoutSeconds < 10 || c.CollectTimeoutSeconds > 300 || c.MaxArtifactBytes < 1 || c.MaxArtifactBytes > 1<<30 {
|
||||
return c, errors.New("invalid worker execution limits")
|
||||
}
|
||||
if len(c.Owners) > 0 && !c.WatchdogRequired {
|
||||
return c, errors.New("independent watchdog is required for configured execution")
|
||||
}
|
||||
for _, b := range c.Owners {
|
||||
if b.MaxOwnedArtifactBytes <= 0 {
|
||||
return c, errors.New("explicit owner private artifact quota required")
|
||||
}
|
||||
}
|
||||
pools := map[string]bool{}
|
||||
storage := map[string]bool{}
|
||||
for owner, b := range c.Owners {
|
||||
if !uuidRE.MatchString(owner) || !ident.MatchString(b.Node) || !ident.MatchString(b.Pool) || !ident.MatchString(b.ISOStorage) || !ident.MatchString(b.DiskStorage) || b.ISOStorage == "local" || b.DiskStorage == "tank-store" {
|
||||
return c, errors.New("invalid private owner binding; shared lab storage forbidden")
|
||||
}
|
||||
if pools[b.Pool] || storage[b.ISOStorage] {
|
||||
return c, errors.New("owner pools and ISO storages must be distinct")
|
||||
}
|
||||
pools[b.Pool] = true
|
||||
storage[b.ISOStorage] = true
|
||||
files := map[string]bool{}
|
||||
for _, p := range []string{b.ProvisionerFile, b.RuntimeFile, b.RecorderFile, b.UploaderFile, b.HousekeepingFile} {
|
||||
if !filepath.IsAbs(p) || files[p] {
|
||||
return c, errors.New("five distinct absolute credential file paths required")
|
||||
}
|
||||
files[p] = true
|
||||
}
|
||||
if b.MinStorageFreeBytes <= 0 || b.MinMemoryFreeBytes <= 0 || b.MaxOwnedDiskBytes <= 0 {
|
||||
return c, errors.New("explicit owner resource headroom and disk quota required")
|
||||
}
|
||||
}
|
||||
for ref, s := range c.Sources {
|
||||
if !ident.MatchString(ref) || !ident.MatchString(s.Node) || s.VMID < 100 || s.VMID == 7000 || !uuidRE.MatchString(s.OwnerID) || !filepath.IsAbs(s.SealPath) || !regexp.MustCompile(`^(ide|sata|scsi)[0-9]+$`).MatchString(s.CDSlot) || s.MaxDiskBytes <= 0 {
|
||||
return c, errors.New("invalid source mapping")
|
||||
}
|
||||
if _, ok := c.Owners[s.OwnerID]; !ok {
|
||||
return c, errors.New("source validation owner missing")
|
||||
}
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
func (b Binding) isolationProof(owner string) (IsolationProof, error) {
|
||||
var p IsolationProof
|
||||
raw, err := os.ReadFile(b.IsolationProofFile)
|
||||
if err != nil || json.Unmarshal(raw, &p) != nil {
|
||||
return p, errors.New("required scoped ACL/storage isolation probe evidence missing")
|
||||
}
|
||||
if !p.Passed || !p.ExpiresAt.After(time.Now()) || p.OwnerID != owner || p.Endpoint != b.Endpoint || p.Node != b.Node || p.Pool != b.Pool || p.ISOStorage != b.ISOStorage || p.DiskStorage != b.DiskStorage || !shaRE.MatchString(p.EvidenceSHA256) {
|
||||
return p, errors.New("scoped isolation proof invalid or expired")
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (b Binding) proof(owner string) error {
|
||||
_, err := b.isolationProof(owner)
|
||||
return err
|
||||
}
|
||||
|
||||
type clients struct{ provisioner, runtime, recorder, uploader, housekeeping *pve.Client }
|
||||
|
||||
func (b Binding) clients() (clients, error) {
|
||||
var c clients
|
||||
for _, p := range []struct {
|
||||
dst **pve.Client
|
||||
file string
|
||||
}{{&c.provisioner, b.ProvisionerFile}, {&c.runtime, b.RuntimeFile}, {&c.recorder, b.RecorderFile}, {&c.uploader, b.UploaderFile}, {&c.housekeeping, b.HousekeepingFile}} {
|
||||
v, err := pve.New(b.Endpoint, b.CAFile, p.file)
|
||||
if err != nil {
|
||||
c.close()
|
||||
return c, err
|
||||
}
|
||||
*p.dst = v
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
func (c clients) authenticate(ctx context.Context) error {
|
||||
for _, credential := range []struct {
|
||||
purpose string
|
||||
client *pve.Client
|
||||
}{{"provisioner", c.provisioner}, {"runtime", c.runtime}, {"recorder", c.recorder}, {"uploader", c.uploader}, {"housekeeping", c.housekeeping}} {
|
||||
var permissions map[string]any
|
||||
if err := credential.client.Do(ctx, "GET", "/access/permissions", nil, &permissions); err != nil {
|
||||
return fmt.Errorf("%s PVE authentication failed: %w", credential.purpose, err)
|
||||
}
|
||||
if permissions == nil {
|
||||
return fmt.Errorf("%s PVE permissions response invalid", credential.purpose)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (c clients) close() {
|
||||
for _, v := range []*pve.Client{c.provisioner, c.runtime, c.recorder, c.uploader, c.housekeeping} {
|
||||
if v != nil {
|
||||
v.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
func (c Config) protected() []int {
|
||||
ids := []int{7000, 7001}
|
||||
for _, s := range c.Sources {
|
||||
ids = append(ids, s.VMID)
|
||||
}
|
||||
if c.Extractor != nil {
|
||||
ids = append(ids, c.Extractor.SourceVMID)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
func contained(root, key string) (string, error) {
|
||||
if filepath.IsAbs(key) || key == "" {
|
||||
return "", errors.New("invalid private storage key")
|
||||
}
|
||||
p := filepath.Join(root, filepath.FromSlash(key))
|
||||
rel, err := filepath.Rel(root, p)
|
||||
if err != nil || rel == ".." || len(rel) > 3 && rel[:3] == ".."+string(filepath.Separator) {
|
||||
return "", errors.New("private storage path escapes root")
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"otche/internal/pve"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestBindingAuthenticationRejectsRevokedPurposeCredential(t *testing.T) {
|
||||
revoked := ""
|
||||
server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/api2/json/access/permissions" || r.Method != "GET" {
|
||||
t.Errorf("unexpected authentication request %s %s", r.Method, r.URL.Path)
|
||||
}
|
||||
if revoked != "" && strings.Contains(r.Header.Get("Authorization"), "!"+revoked+"=") {
|
||||
http.Error(w, "authentication failed", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{}})
|
||||
}))
|
||||
defer server.Close()
|
||||
dir := t.TempDir()
|
||||
ca := filepath.Join(dir, "ca.pem")
|
||||
if err := os.WriteFile(ca, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw}), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var cs clients
|
||||
for _, slot := range []struct {
|
||||
purpose string
|
||||
client **pve.Client
|
||||
}{{"provisioner", &cs.provisioner}, {"runtime", &cs.runtime}, {"recorder", &cs.recorder}, {"uploader", &cs.uploader}, {"housekeeping", &cs.housekeeping}} {
|
||||
file := filepath.Join(dir, slot.purpose+".json")
|
||||
if err := atomicJSON(file, pve.Credential{TokenID: "otche@pve!" + slot.purpose, Secret: "test-only"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var err error
|
||||
*slot.client, err = pve.New(server.URL, ca, file)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
defer cs.close()
|
||||
if err := cs.authenticate(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, purpose := range []string{"provisioner", "runtime", "recorder", "uploader", "housekeeping"} {
|
||||
revoked = purpose
|
||||
if err := cs.authenticate(context.Background()); err == nil || !strings.Contains(err.Error(), purpose) {
|
||||
t.Fatalf("revoked %s credential did not block binding authentication: %v", purpose, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
//go:build !windows
|
||||
|
||||
package worker
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
func artifactHeadroom(path string, required int64) error {
|
||||
var stat syscall.Statfs_t
|
||||
if err := syscall.Statfs(path, &stat); err != nil {
|
||||
return err
|
||||
}
|
||||
if uint64(stat.Bavail)*uint64(stat.Bsize) < uint64(required) {
|
||||
return errors.New("private artifact filesystem headroom insufficient")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"syscall"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
func artifactHeadroom(path string, required int64) error {
|
||||
p, err := syscall.UTF16PtrFromString(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var free, total, totalFree uint64
|
||||
proc := syscall.NewLazyDLL("kernel32.dll").NewProc("GetDiskFreeSpaceExW")
|
||||
ok, _, callErr := proc.Call(uintptr(unsafe.Pointer(p)), uintptr(unsafe.Pointer(&free)), uintptr(unsafe.Pointer(&total)), uintptr(unsafe.Pointer(&totalFree)))
|
||||
if ok == 0 {
|
||||
return callErr
|
||||
}
|
||||
if free < uint64(required) {
|
||||
return errors.New("private artifact filesystem headroom insufficient")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,654 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"otche/internal/grub"
|
||||
"otche/internal/pve"
|
||||
"otche/internal/recorder"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type guestReady struct {
|
||||
Ready bool `json:"ready"`
|
||||
SessionID int `json:"session_id"`
|
||||
User string `json:"user"`
|
||||
Privilege string `json:"privilege"`
|
||||
BootID string `json:"boot_id"`
|
||||
Baseline struct {
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
Qualified bool `json:"qualified"`
|
||||
Errors []string `json:"errors"`
|
||||
} `json:"baseline"`
|
||||
Defender json.RawMessage `json:"defender"`
|
||||
Environment json.RawMessage `json:"environment"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
type receipt struct {
|
||||
CommandID string `json:"command_id"`
|
||||
State string `json:"state"`
|
||||
StartedAt *time.Time `json:"started_at"`
|
||||
DeadlineAt *time.Time `json:"deadline_at"`
|
||||
BootID string `json:"boot_id"`
|
||||
SessionID int `json:"session_id"`
|
||||
PID int `json:"pid"`
|
||||
Privilege string `json:"privilege"`
|
||||
}
|
||||
type guestResult struct {
|
||||
CommandID string `json:"command_id"`
|
||||
AttemptID string `json:"attempt_id"`
|
||||
JobID string `json:"job_id"`
|
||||
Phase string `json:"phase"`
|
||||
Outcome string `json:"outcome"`
|
||||
Findings string `json:"findings"`
|
||||
Telemetry string `json:"telemetry"`
|
||||
StartedAt *time.Time `json:"started_at"`
|
||||
FinishedAt *time.Time `json:"finished_at"`
|
||||
Report json.RawMessage `json:"report"`
|
||||
Artifacts []struct {
|
||||
Path string `json:"path"`
|
||||
Kind string `json:"kind"`
|
||||
ContentType string `json:"content_type"`
|
||||
} `json:"artifacts"`
|
||||
}
|
||||
|
||||
func guestDir(a attempt) string { return `C:\ProgramData\Otche\results\` + a.CommandID + `\` }
|
||||
func guestScript(ctx context.Context, c *pve.Client, b Binding, l allocation, name string, args ...string) ([]byte, error) {
|
||||
command := []string{"powershell.exe", "-NoProfile", "-NonInteractive", "-File", `C:\ProgramData\Otche\runner\` + name}
|
||||
command = append(command, args...)
|
||||
pid, err := c.Exec(ctx, b.Node, l.VMID, command)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.ExecWait(ctx, b.Node, l.VMID, pid)
|
||||
}
|
||||
func readGuestJSON(ctx context.Context, c *pve.Client, b Binding, l allocation, path string, v any) error {
|
||||
var raw bytes.Buffer
|
||||
_, err := c.ReadFile(ctx, b.Node, l.VMID, path, &raw, 4<<20)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if json.Unmarshal(raw.Bytes(), v) != nil {
|
||||
return errors.New("guest control file is not valid bounded JSON")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (e *engine) ready(ctx context.Context, a attempt, l allocation, b Binding, cs clients, privilege string) (guestReady, error) {
|
||||
var ready guestReady
|
||||
t := time.NewTicker(2 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
call, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
raw, err := guestScript(call, cs.runtime, b, l, "Test-OtcheReady.ps1", "-Privilege", privilege)
|
||||
cancel()
|
||||
parsed := json.Unmarshal(raw, &ready) == nil
|
||||
if err == nil && parsed && ready.Ready && ready.SessionID > 0 && ready.User != "" && ready.BootID != "" && shaRE.MatchString(ready.Baseline.Fingerprint) {
|
||||
return ready, nil
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
path := filepath.Join(e.root, "attempts", a.ID, "readiness-diagnostic.json")
|
||||
if parsed {
|
||||
_ = atomicJSON(path, ready)
|
||||
publish, c := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
_ = e.publishFile(publish, a, "prerequisite", "readiness-diagnostic.json", "application/json", path)
|
||||
c()
|
||||
}
|
||||
return ready, errors.New("Source prerequisite failed: QGA, active interactive desktop, enabled tasks, active Defender and operator-approved signed runner script policy required; Restricted blocks the runner and is never bypassed. See readiness diagnostic when available.")
|
||||
case <-t.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
func (e *engine) safetyDeadline(ctx context.Context, a attempt, l allocation, deadline time.Time) error {
|
||||
tag, err := e.db.Exec(ctx, "UPDATE allocations SET metadata=metadata || jsonb_build_object('safety_deadline',$2::text) WHERE id=$1 AND EXISTS(SELECT 1 FROM attempts WHERE id=$3 AND lease_owner=$4 AND lease_until>now())", l.ID, deadline.UTC().Format(time.RFC3339Nano), a.ID, e.id)
|
||||
if err == nil && tag.RowsAffected() != 1 {
|
||||
return errors.New("attempt lease lost")
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (e *engine) execute(ctx context.Context, a attempt) (outcome, findings, telemetry, cleanup string, report json.RawMessage, retErr error) {
|
||||
outcome, findings, telemetry, cleanup = "error", "unknown", "unavailable", "complete"
|
||||
if a.Phase != "queued" {
|
||||
cleanup = "evidence_held"
|
||||
}
|
||||
b, ok := e.cfg.Owners[a.OwnerID]
|
||||
if !ok {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("owner credentials are not configured")
|
||||
}
|
||||
cs, err := b.clients()
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
defer cs.close()
|
||||
var opts settings
|
||||
var internal struct {
|
||||
Qualification string `json:"_qualification"`
|
||||
}
|
||||
if json.Unmarshal(a.Settings, &opts) != nil || json.Unmarshal(a.Settings, &internal) != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("invalid job settings")
|
||||
}
|
||||
if opts.Internet != "offline" && (opts.Internet != "online" || b.Online == nil) {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("online execution requires configured qualified isolated policy")
|
||||
}
|
||||
if opts.Duration < 30 || opts.Duration > 1200 || (opts.Privilege != "user" && opts.Privilege != "admin") {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("invalid immutable run limits")
|
||||
}
|
||||
if err = grub.Validate(opts.GrubPaths); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
manifest := map[string]any{"command_id": a.CommandID, "attempt_id": a.ID, "job_id": a.JobID, "iso_label": isoLabel(a.JobID), "sha256": a.SHA256, "filename": a.Filename, "settings": json.RawMessage(a.Settings)}
|
||||
if len(opts.GrubPaths) > 0 {
|
||||
manifest["grub_limits"] = map[string]any{"total_bytes": min(int64(32<<20), e.cfg.MaxArtifactBytes/2), "file_bytes": min(int64(8<<20), e.cfg.MaxArtifactBytes/2), "seconds": max(1, e.cfg.CollectTimeoutSeconds/2)}
|
||||
}
|
||||
if internal.Qualification != "" {
|
||||
manifest["qualification"] = internal.Qualification
|
||||
}
|
||||
control, _ := json.Marshal(manifest)
|
||||
if len(control) > pve.ControlLimit {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("serialized guest manifest exceeds transport bound before allocation")
|
||||
}
|
||||
dir := filepath.Join(e.root, "attempts", a.ID)
|
||||
if err = os.MkdirAll(dir, 0700); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = artifactHeadroom(e.root, e.cfg.MinArtifactFreeBytes+e.cfg.MaxVideoBytes+grubReservation(a, e.cfg.MaxArtifactBytes)); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
var ownedBytes int64
|
||||
if err = e.db.QueryRow(ctx, `SELECT COALESCE(sum(bytes),0) FROM (SELECT size bytes FROM uploads WHERE owner_id=$1 UNION ALL SELECT size FROM artifacts WHERE job_id IN (SELECT id FROM jobs WHERE owner_id=$1) UNION ALL SELECT u.size+4194304 FROM media m JOIN jobs j ON j.id=m.job_id JOIN uploads u ON u.id=j.upload_id WHERE m.owner_id=$1 UNION ALL SELECT COALESCE((metadata->>'artifact_reservation')::bigint,0) FROM allocations WHERE owner_id=$1 AND state<>'deleted') usage`, a.OwnerID).Scan(&ownedBytes); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if a.Phase == "queued" && (b.MaxOwnedArtifactBytes <= 0 || ownedBytes+3*e.cfg.MaxArtifactBytes+e.cfg.MaxVideoBytes+grubReservation(a, e.cfg.MaxArtifactBytes)+a.Size+4194304 > b.MaxOwnedArtifactBytes) {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("owner input/media/artifact reservation quota exceeded including retained evidence")
|
||||
}
|
||||
var l allocation
|
||||
var session *recorder.Session
|
||||
recordingDir := filepath.Join(dir, "video")
|
||||
videoComplete := false
|
||||
collected := false
|
||||
defer func() {
|
||||
finalSeconds := e.cfg.CollectTimeoutSeconds + 120
|
||||
if e.cfg.Extractor != nil {
|
||||
finalSeconds += e.cfg.Extractor.TimeoutSeconds
|
||||
}
|
||||
final, cancel := context.WithTimeout(context.Background(), time.Duration(finalSeconds)*time.Second)
|
||||
defer cancel()
|
||||
if err := e.leaseAlive(final, a); err != nil {
|
||||
if session != nil {
|
||||
_ = session.Close()
|
||||
}
|
||||
cleanup = "evidence_held"
|
||||
retErr = err
|
||||
return
|
||||
}
|
||||
if l.ID != "" {
|
||||
_ = e.safetyDeadline(final, a, l, time.Now().Add(time.Duration(finalSeconds)*time.Second))
|
||||
}
|
||||
if l.ID != "" && !collected {
|
||||
salvage, done := context.WithTimeout(final, 15*time.Second)
|
||||
partial, detected := e.salvagePartial(salvage, a, l, b, cs, dir)
|
||||
if partial != nil {
|
||||
report = partial
|
||||
telemetry = "partial"
|
||||
}
|
||||
if detected == "detected" {
|
||||
findings = detected
|
||||
}
|
||||
done()
|
||||
}
|
||||
if session != nil {
|
||||
if err := session.Close(); err != nil {
|
||||
videoComplete = false
|
||||
if retErr == nil {
|
||||
retErr = errors.New("recorder did not finalize completely")
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := e.publishVideo(final, a, recordingDir, videoComplete); err != nil {
|
||||
videoComplete = false
|
||||
if retErr == nil {
|
||||
retErr = err
|
||||
}
|
||||
}
|
||||
journal := filepath.Join(dir, "observations.jsonl")
|
||||
if _, err := os.Stat(journal); err == nil {
|
||||
if err = e.publishFile(final, a, "observation_journal", "observations.jsonl", "application/x-ndjson", journal); err != nil {
|
||||
collected = false
|
||||
if retErr == nil {
|
||||
retErr = err
|
||||
}
|
||||
}
|
||||
}
|
||||
if l.ID == "" {
|
||||
existing, err := e.getAllocation(final, a)
|
||||
if err == nil {
|
||||
l = existing
|
||||
}
|
||||
}
|
||||
if l.ID != "" {
|
||||
cleanup = "evidence_held"
|
||||
stopErr := e.stopOwned(final, a, l, b, cs)
|
||||
if stopErr != nil {
|
||||
cleanup = "failed"
|
||||
if retErr == nil {
|
||||
retErr = stopErr
|
||||
}
|
||||
} else if collected && videoComplete {
|
||||
if err := e.deleteOwned(final, a, l, b, cs); err != nil {
|
||||
cleanup = "failed"
|
||||
if retErr == nil {
|
||||
retErr = err
|
||||
}
|
||||
} else {
|
||||
cleanup = "complete"
|
||||
}
|
||||
} else {
|
||||
_, _ = e.db.Exec(final, "UPDATE allocations SET state='evidence_held' WHERE id=$1", l.ID)
|
||||
if e.cfg.Extractor != nil {
|
||||
if err := e.extractHeld(final, a, l, b, cs, dir); err != nil {
|
||||
e.event(final, a, "retention", err.Error())
|
||||
}
|
||||
}
|
||||
e.event(final, a, "retention", "Collection incomplete. Stopped clone and evidence retained; explicit administrative release required.")
|
||||
}
|
||||
if cleanup == "complete" {
|
||||
if err := e.removeMedia(final, a, b, cs); err != nil {
|
||||
cleanup = "failed"
|
||||
if retErr == nil {
|
||||
retErr = err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if recovered, observed := salvageExtracted(dir, a); recovered != nil {
|
||||
report = recovered
|
||||
telemetry = "partial"
|
||||
if observed == "detected" {
|
||||
findings = observed
|
||||
}
|
||||
}
|
||||
if !videoComplete && telemetry == "complete" {
|
||||
telemetry = "partial"
|
||||
}
|
||||
manifest := map[string]any{"attempt_id": a.ID, "job_id": a.JobID, "command_id": a.CommandID, "source_revision": a.RevisionID, "original_sha256": a.SHA256, "outcome": outcome, "findings": findings, "telemetry": telemetry, "cleanup": cleanup, "finished_at": time.Now().UTC(), "guest_evidence_trust": "Guest administrator can tamper with guest telemetry; external video has independent provenance."}
|
||||
if retErr != nil {
|
||||
manifest["error"] = retErr.Error()
|
||||
}
|
||||
manifestPath := filepath.Join(dir, "manifest.json")
|
||||
if err := atomicJSON(manifestPath, manifest); err == nil {
|
||||
if err = e.publishFile(final, a, "manifest", "manifest.json", "application/json", manifestPath); err != nil && retErr == nil {
|
||||
retErr = err
|
||||
}
|
||||
}
|
||||
}()
|
||||
if err = e.fence(ctx, a); err != nil {
|
||||
outcome = "cancelled"
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if a.Phase != "queued" {
|
||||
outcome = "interrupted"
|
||||
l, err = e.getAllocation(ctx, a)
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if l.UPID != nil {
|
||||
wait, cancel := context.WithTimeout(ctx, time.Duration(e.cfg.TaskTimeoutSeconds)*time.Second)
|
||||
err = cs.provisioner.Task(wait, b.Node, *l.UPID)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("recovery cannot confirm recorded PVE task; no re-execution")
|
||||
}
|
||||
}
|
||||
// A restarted recorder cannot recreate the missed interval. Recover available
|
||||
// guest evidence, but never relaunch a accepted/unknown command or claim completeness.
|
||||
recovery, cancel := context.WithTimeout(ctx, time.Duration(e.cfg.CollectTimeoutSeconds)*time.Second)
|
||||
defer cancel()
|
||||
var r guestResult
|
||||
if readGuestJSON(recovery, cs.runtime, b, l, guestDir(a)+"result.json", &r) == nil && validResult(a, r) {
|
||||
report = r.Report
|
||||
findings = r.Findings
|
||||
telemetry = "partial"
|
||||
_ = e.collect(recovery, a, l, b, cs, r, dir)
|
||||
}
|
||||
return outcome, findings, telemetry, cleanup, report, errors.New("worker restart interrupted observation; durable command was not relaunched")
|
||||
}
|
||||
if err = b.proof(a.OwnerID); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if internal.Qualification == "" {
|
||||
var baseline struct {
|
||||
State string `json:"state"`
|
||||
Baseline string `json:"baseline_fingerprint"`
|
||||
}
|
||||
if json.Unmarshal(a.Qualification, &baseline) != nil || baseline.State != "qualified" || !shaRE.MatchString(baseline.Baseline) {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("revision has no observed successful qualification baseline")
|
||||
}
|
||||
}
|
||||
volume, err := e.media(ctx, a, b, cs)
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = e.phase(ctx, a, "provisioning"); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
l, err = e.provision(ctx, a, b, cs)
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
prep, cancelPrep := context.WithTimeout(ctx, time.Duration(e.cfg.PrepareTimeoutSeconds)*time.Second)
|
||||
defer cancelPrep()
|
||||
if err = e.safetyDeadline(ctx, a, l, time.Now().Add(time.Duration(e.cfg.PrepareTimeoutSeconds)*time.Second)); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = e.prepareClone(prep, a, l, b, cs); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = e.phase(ctx, a, "booting"); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
upid, err := cs.provisioner.OwnedTask(prep, e.own(a, l, b), "POST", "/status/start", nil)
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = e.recordAction(ctx, a, l, "booting", upid); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = cs.provisioner.Task(prep, b.Node, upid); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
ready, err := e.ready(prep, a, l, b, cs, opts.Privilege)
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if len(ready.Baseline.Errors) > 0 {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("Defender baseline prerequisites failed")
|
||||
}
|
||||
if internal.Qualification == "" {
|
||||
var baseline struct {
|
||||
Baseline string `json:"baseline_fingerprint"`
|
||||
}
|
||||
json.Unmarshal(a.Qualification, &baseline)
|
||||
if ready.Baseline.Fingerprint != baseline.Baseline {
|
||||
_, _ = e.db.Exec(ctx, "UPDATE profiles SET qualification='drifted',enabled=false,reason='Observed Defender baseline drift' WHERE id=$1", a.ProfileID)
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("Defender baseline drift detected before delivery")
|
||||
}
|
||||
}
|
||||
baselinePath := filepath.Join(dir, "baseline.json")
|
||||
if err = atomicJSON(baselinePath, ready); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = e.publishFile(ctx, a, "baseline", "baseline.json", "application/json", baselinePath); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = e.phase(ctx, a, "recording"); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = cs.provisioner.CheckOwned(prep, e.own(a, l, b)); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
ws, ticket, err := cs.recorder.Console(prep, e.own(a, l, b))
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
session, err = recorder.Start(ctx, recorder.Config{URL: ws, Header: cs.recorder.Header(), TLSConfig: cs.recorder.TLSConfig(), Ticket: ticket, Directory: recordingDir, FPS: 10, MaxBytes: e.cfg.MaxVideoBytes})
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
videoComplete = true
|
||||
_, _ = e.db.Exec(ctx, "UPDATE attempts SET video=jsonb_build_object('state','recording','segments','[]'::jsonb,'gaps','[]'::jsonb,'started_at',now(),'finished_at',null) WHERE id=$1", a.ID)
|
||||
select {
|
||||
case <-session.Done():
|
||||
videoComplete = false
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("recorder lost before delivery")
|
||||
default:
|
||||
}
|
||||
delivery, cancelDelivery := context.WithTimeout(ctx, time.Duration(e.cfg.PrepareTimeoutSeconds)*time.Second)
|
||||
defer cancelDelivery()
|
||||
if err = e.safetyDeadline(ctx, a, l, time.Now().Add(time.Duration(e.cfg.PrepareTimeoutSeconds)*time.Second)); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = e.phase(ctx, a, "delivering"); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = e.swapISO(delivery, a, l, b, cs, volume); err != nil {
|
||||
outcome = "delivery_error"
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = session.Err(); err != nil {
|
||||
videoComplete = false
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("recorder failed during ISO insertion; dispatch prohibited")
|
||||
}
|
||||
controlPath := `C:\ProgramData\Otche\control\` + a.CommandID + ".json"
|
||||
if err = cs.runtime.WriteControl(delivery, b.Node, l.VMID, controlPath, control); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
var verify bytes.Buffer
|
||||
if _, err = cs.runtime.ReadFile(delivery, b.Node, l.VMID, controlPath, &verify, pve.ControlLimit+1); err != nil || !bytes.Equal(control, verify.Bytes()) {
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("bounded guest manifest verification failed")
|
||||
}
|
||||
if err = session.Err(); err != nil {
|
||||
videoComplete = false
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("recorder failed before run permit; dispatch prohibited")
|
||||
}
|
||||
select {
|
||||
case <-session.Done():
|
||||
videoComplete = false
|
||||
return outcome, findings, telemetry, cleanup, nil, errors.New("recorder closed before run permit")
|
||||
default:
|
||||
}
|
||||
if err = e.recordAction(ctx, a, l, "dispatch_intent", ""); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
if err = e.phase(ctx, a, "preparing"); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
// Dispatch once only. Even an HTTP timeout can mean the scheduler accepted it.
|
||||
dispatch, cancelDispatch := context.WithTimeout(delivery, 30*time.Second)
|
||||
_, dispatchErr := guestScript(dispatch, cs.runtime, b, l, "Start-OtcheCommand.ps1", "-ManifestPath", controlPath)
|
||||
cancelDispatch()
|
||||
if dispatchErr != nil {
|
||||
e.event(ctx, a, "dispatch", "Scheduler response unavailable; reconciling durable receipt without relaunch")
|
||||
}
|
||||
r, err := e.observe(ctx, delivery, a, l, b, cs, session, ready, opts, dir)
|
||||
if err != nil {
|
||||
videoComplete = false
|
||||
outcome = "interrupted"
|
||||
if ctx.Err() != nil {
|
||||
var cancelled bool
|
||||
check, done := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
_ = e.db.QueryRow(check, "SELECT cancel_requested FROM jobs WHERE id=$1", a.JobID).Scan(&cancelled)
|
||||
done()
|
||||
if cancelled {
|
||||
outcome = "cancelled"
|
||||
}
|
||||
}
|
||||
return outcome, findings, telemetry, cleanup, nil, err
|
||||
}
|
||||
outcome, findings, telemetry, report = r.Outcome, r.Findings, r.Telemetry, r.Report
|
||||
collectCtx, cancelCollect := context.WithTimeout(ctx, time.Duration(e.cfg.CollectTimeoutSeconds)*time.Second)
|
||||
defer cancelCollect()
|
||||
if err = e.phase(collectCtx, a, "collecting"); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, report, err
|
||||
}
|
||||
if err = e.collect(collectCtx, a, l, b, cs, r, dir); err != nil {
|
||||
telemetry = "partial"
|
||||
return outcome, findings, telemetry, cleanup, report, err
|
||||
}
|
||||
if len(opts.GrubPaths) > 0 {
|
||||
report, err = e.collectGrub(collectCtx, a, l, b, cs, opts.GrubPaths, report, dir)
|
||||
if err != nil {
|
||||
return outcome, findings, telemetry, cleanup, report, err
|
||||
}
|
||||
}
|
||||
rawAfter, err := guestScript(collectCtx, cs.runtime, b, l, "Test-OtcheReady.ps1", "-Privilege", opts.Privilege, "-BaselineOnly")
|
||||
if err != nil {
|
||||
telemetry = "partial"
|
||||
return outcome, findings, telemetry, cleanup, report, errors.New("Final full source/Defender baseline unavailable")
|
||||
}
|
||||
var after guestReady
|
||||
if json.Unmarshal(rawAfter, &after) != nil || after.Baseline.Fingerprint != ready.Baseline.Fingerprint {
|
||||
telemetry = "partial"
|
||||
_, _ = e.db.Exec(ctx, "UPDATE profiles SET qualification='drifted',enabled=false,reason='Full baseline changed during observation' WHERE id=$1", a.ProfileID)
|
||||
return outcome, findings, telemetry, cleanup, report, errors.New("Full source/Defender baseline drift during observation")
|
||||
}
|
||||
afterPath := filepath.Join(dir, "baseline-after.json")
|
||||
if err = atomicJSON(afterPath, after); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, report, err
|
||||
}
|
||||
if err = e.publishFile(ctx, a, "baseline_after", "baseline-after.json", "application/json", afterPath); err != nil {
|
||||
return outcome, findings, telemetry, cleanup, report, err
|
||||
}
|
||||
collected = r.Telemetry == "complete"
|
||||
return outcome, findings, telemetry, cleanup, report, nil
|
||||
}
|
||||
func validResult(a attempt, r guestResult) bool {
|
||||
return r.CommandID == a.CommandID && r.AttemptID == a.ID && r.JobID == a.JobID && strings.Contains("|executed|blocked_before_execution|incompatible|policy_blocked|delivery_error|interrupted|cancelled|error|", "|"+r.Outcome+"|") && strings.Contains("|unknown|detected|not_observed|", "|"+r.Findings+"|") && strings.Contains("|complete|partial|unavailable|", "|"+r.Telemetry+"|") && json.Valid(r.Report)
|
||||
}
|
||||
func (e *engine) observe(ctx, preparing context.Context, a attempt, l allocation, b Binding, cs clients, session *recorder.Session, ready guestReady, opts settings, dir string) (guestResult, error) {
|
||||
t := time.NewTicker(2 * time.Second)
|
||||
defer t.Stop()
|
||||
var actualEnd *time.Time
|
||||
lastRead := time.Now()
|
||||
lastBootCheck := time.Time{}
|
||||
var observed receipt
|
||||
for {
|
||||
if err := e.fence(ctx, a); err != nil {
|
||||
return guestResult{}, err
|
||||
}
|
||||
select {
|
||||
case <-session.Done():
|
||||
return guestResult{}, errors.New("external video observation lost")
|
||||
default:
|
||||
}
|
||||
call, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
var rec receipt
|
||||
receiptErr := readGuestJSON(call, cs.runtime, b, l, guestDir(a)+"receipt.json", &rec)
|
||||
if receiptErr == nil {
|
||||
if rec.CommandID != a.CommandID {
|
||||
cancel()
|
||||
return guestResult{}, errors.New("guest receipt command mismatch")
|
||||
}
|
||||
lastRead = time.Now()
|
||||
observed = rec
|
||||
if rec.StartedAt != nil && actualEnd == nil {
|
||||
if rec.SessionID != ready.SessionID || rec.BootID != ready.BootID || rec.PID <= 0 || rec.Privilege != opts.Privilege || rec.StartedAt.After(time.Now().Add(5*time.Second)) || rec.StartedAt.Before(time.Now().Add(-45*time.Second)) {
|
||||
cancel()
|
||||
return guestResult{}, errors.New("runner receipt context or start time is invalid")
|
||||
}
|
||||
end := rec.StartedAt.Add(time.Duration(opts.Duration) * time.Second)
|
||||
actualEnd = &end
|
||||
_, err := e.db.Exec(ctx, "UPDATE attempts SET phase='running',started_at=$2,deadline_at=$3 WHERE id=$1 AND lease_owner=$4", a.ID, *rec.StartedAt, end, e.id)
|
||||
if err != nil {
|
||||
cancel()
|
||||
return guestResult{}, err
|
||||
}
|
||||
if err = e.safetyDeadline(ctx, a, l, end.Add(time.Duration(e.cfg.CollectTimeoutSeconds)*time.Second)); err != nil {
|
||||
cancel()
|
||||
return guestResult{}, err
|
||||
}
|
||||
}
|
||||
}
|
||||
var result guestResult
|
||||
resultErr := readGuestJSON(call, cs.runtime, b, l, guestDir(a)+"result.json", &result)
|
||||
if resultErr == nil {
|
||||
if !validResult(a, result) {
|
||||
cancel()
|
||||
return result, errors.New("guest result identity or outcome contract invalid")
|
||||
}
|
||||
lastRead = time.Now()
|
||||
if actualEnd == nil && result.StartedAt == nil {
|
||||
cancel()
|
||||
if result.Outcome == "executed" {
|
||||
return result, errors.New("execution result lacks actual runner receipt")
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
if actualEnd != nil && time.Now().After(*actualEnd) {
|
||||
cancel()
|
||||
return result, nil
|
||||
}
|
||||
}
|
||||
var status json.RawMessage
|
||||
if readGuestJSON(call, cs.runtime, b, l, guestDir(a)+"status.json", &status) == nil {
|
||||
lastRead = time.Now()
|
||||
if err := appendObservation(filepath.Join(dir, "observations.jsonl"), status, e.cfg.MaxArtifactBytes/2); err != nil {
|
||||
cancel()
|
||||
return guestResult{}, err
|
||||
}
|
||||
if err := atomicJSON(filepath.Join(dir, "status-latest.json"), status); err != nil {
|
||||
cancel()
|
||||
return guestResult{}, err
|
||||
}
|
||||
}
|
||||
if actualEnd != nil && time.Since(lastBootCheck) > 10*time.Second {
|
||||
lastBootCheck = time.Now()
|
||||
raw, err := guestScript(call, cs.runtime, b, l, "Test-OtcheReady.ps1", "-Privilege", opts.Privilege, "-BootOnly")
|
||||
if err == nil {
|
||||
var now guestReady
|
||||
if json.Unmarshal(raw, &now) == nil && now.BootID != "" && now.BootID != observed.BootID {
|
||||
cancel()
|
||||
return guestResult{}, errors.New("guest reboot interrupted attempt; no resume")
|
||||
}
|
||||
}
|
||||
}
|
||||
cancel()
|
||||
if time.Since(lastRead) > 30*time.Second {
|
||||
return guestResult{}, errors.New("guest agent telemetry watchdog expired")
|
||||
}
|
||||
if actualEnd == nil && preparing.Err() != nil {
|
||||
return guestResult{}, errors.New("runner never established actual-start receipt before preparation deadline")
|
||||
}
|
||||
if actualEnd != nil && time.Now().After(actualEnd.Add(time.Duration(e.cfg.CollectTimeoutSeconds)*time.Second)) {
|
||||
return guestResult{}, errors.New("external run and collection deadline expired")
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return guestResult{}, ctx.Err()
|
||||
case <-t.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
func (e *engine) collect(ctx context.Context, a attempt, l allocation, b Binding, cs clients, r guestResult, dir string) error {
|
||||
collectionDir := filepath.Join(dir, "collection-"+newID())
|
||||
if err := os.MkdirAll(collectionDir, 0700); err != nil {
|
||||
return err
|
||||
}
|
||||
paths := []struct{ path, kind, ctype string }{{guestDir(a) + "receipt.json", "receipt", "application/json"}, {guestDir(a) + "status.json", "status", "application/json"}, {guestDir(a) + "result.json", "result", "application/json"}}
|
||||
if len(r.Artifacts) > 64 {
|
||||
return errors.New("guest artifact count exceeds bound")
|
||||
}
|
||||
for _, v := range r.Artifacts {
|
||||
if !strings.HasPrefix(strings.ToLower(v.Path), strings.ToLower(guestDir(a))) || strings.Contains(strings.TrimPrefix(v.Path, guestDir(a)), "..") || strings.ContainsAny(strings.TrimPrefix(v.Path, guestDir(a)), ":/\x00") {
|
||||
return errors.New("guest artifact path not within this command result directory")
|
||||
}
|
||||
paths = append(paths, struct{ path, kind, ctype string }{v.Path, v.Kind, v.ContentType})
|
||||
}
|
||||
var total int64
|
||||
for i, v := range paths {
|
||||
path := filepath.Join(collectionDir, fmt.Sprintf("guest-%03d-%s", i, filepath.Base(strings.ReplaceAll(v.path, "\\", "/"))))
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
n, err := cs.runtime.ReadFile(ctx, b.Node, l.VMID, v.path, f, e.cfg.MaxArtifactBytes-total)
|
||||
total += n
|
||||
syncErr := f.Sync()
|
||||
closeErr := f.Close()
|
||||
if err != nil || syncErr != nil || closeErr != nil {
|
||||
return errors.New("guest artifact collection incomplete; original evidence retained")
|
||||
}
|
||||
if err = e.publishFile(ctx, a, v.kind, filepath.Base(path), v.ctype, path); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (e *engine) event(ctx context.Context, a attempt, kind, message string) {
|
||||
_, _ = e.db.Exec(ctx, "INSERT INTO events(job_id,attempt_id,kind,message) VALUES($1,$2,$3,$4)", a.JobID, a.ID, kind, message)
|
||||
}
|
||||
@@ -0,0 +1,603 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"otche/internal/pve"
|
||||
)
|
||||
|
||||
// ExtractorConfig names an operator-qualified ordinary Linux source, never a
|
||||
// host directory. Its installed fixed collector parses NTFS only in a disposable VM.
|
||||
type ExtractorConfig struct {
|
||||
Node string `json:"node"`
|
||||
SourceVMID int `json:"source_vmid"`
|
||||
ConfigDigest string `json:"config_digest"`
|
||||
ProofFile string `json:"proof_file"`
|
||||
SourceDisk string `json:"source_disk"`
|
||||
MaxDiskBytes int64 `json:"max_disk_bytes"`
|
||||
IncludeCrashDump bool `json:"include_crash_dump"`
|
||||
TimeoutSeconds int `json:"timeout_seconds"`
|
||||
}
|
||||
type extractionAllocation struct {
|
||||
ID string
|
||||
VMID int
|
||||
State string
|
||||
UPID *string
|
||||
DiskSlot, Serial string
|
||||
DiskBytes int64
|
||||
}
|
||||
type extractionProof struct {
|
||||
Passed bool `json:"passed"`
|
||||
Node string `json:"node"`
|
||||
SourceVMID int `json:"source_vmid"`
|
||||
ConfigDigest string `json:"config_digest"`
|
||||
ExpiresAt time.Time `json:"expires_at"`
|
||||
EvidenceSHA256 string `json:"evidence_sha256"`
|
||||
ReadOnlyVerified bool `json:"read_only_verified"`
|
||||
NoEgress bool `json:"no_egress"`
|
||||
}
|
||||
|
||||
var diskSlotRE = regexp.MustCompile(`^scsi[0-9]+$`)
|
||||
var volumeSizeRE = regexp.MustCompile(`(?:^|,)size=([0-9]+)([KMGTP]?)(?:,|$)`)
|
||||
|
||||
func diskSize(value string) (int64, error) {
|
||||
m := volumeSizeRE.FindStringSubmatch(value)
|
||||
if m == nil {
|
||||
return 0, errors.New("evidence disk size absent")
|
||||
}
|
||||
n, e := strconv.ParseInt(m[1], 10, 64)
|
||||
if e != nil || n <= 0 {
|
||||
return 0, errors.New("invalid evidence size")
|
||||
}
|
||||
power := strings.Index("KMGTP", m[2]) + 1
|
||||
if m[2] == "" {
|
||||
power = 0
|
||||
}
|
||||
for range power {
|
||||
if n > 1<<50 {
|
||||
return 0, errors.New("evidence disk too large")
|
||||
}
|
||||
n *= 1024
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
func diskOption(value, key, want string) string {
|
||||
parts := strings.Split(value, ",")
|
||||
out := []string{parts[0]}
|
||||
for _, p := range parts[1:] {
|
||||
if !strings.HasPrefix(p, key+"=") {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return strings.Join(append(out, key+"="+want), ",")
|
||||
}
|
||||
func diskHas(value, key, want string) bool {
|
||||
for _, p := range strings.Split(value, ",") {
|
||||
if p == key+"="+want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
func cdromMatches(value, volume string) bool {
|
||||
actual, options, ok := strings.Cut(value, ",")
|
||||
if !ok || volume == "" || actual != volume {
|
||||
return false
|
||||
}
|
||||
media := false
|
||||
for _, option := range strings.Split(options, ",") {
|
||||
key, value, found := strings.Cut(option, "=")
|
||||
if !found || key == "" || value == "" || key == "file" || key == "volume" {
|
||||
return false
|
||||
}
|
||||
if key == "media" {
|
||||
if media || value != "cdrom" {
|
||||
return false
|
||||
}
|
||||
media = true
|
||||
}
|
||||
}
|
||||
return media
|
||||
}
|
||||
func ownVolume(value, storage string, id int) bool {
|
||||
volume, _, _ := strings.Cut(value, ",")
|
||||
store, name, found := strings.Cut(volume, ":")
|
||||
if !found || store != storage || id < 1 {
|
||||
return false
|
||||
}
|
||||
vmid := strconv.Itoa(id)
|
||||
if directory, file, isFile := strings.Cut(name, "/"); isFile {
|
||||
if directory != vmid {
|
||||
return false
|
||||
}
|
||||
var format string
|
||||
name, format, found = strings.Cut(file, ".")
|
||||
if !found || (format != "raw" && format != "qcow2" && format != "vmdk") {
|
||||
return false
|
||||
}
|
||||
}
|
||||
disk, found := strings.CutPrefix(name, "vm-"+vmid+"-disk-")
|
||||
if !found || disk == "" {
|
||||
return false
|
||||
}
|
||||
for _, digit := range disk {
|
||||
if digit < '0' || digit > '9' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
func (e *engine) extractOwn(a attempt, x extractionAllocation, b Binding) pve.Ownership {
|
||||
return pve.Ownership{Node: b.Node, VMID: x.VMID, Pool: b.Pool, OwnerID: a.OwnerID, AttemptID: a.ID, Name: "otche-extractor-" + a.ID, Protected: e.cfg.protected()}
|
||||
}
|
||||
func (e *engine) extraction(ctx context.Context, a attempt) (extractionAllocation, error) {
|
||||
var x extractionAllocation
|
||||
err := e.db.QueryRow(ctx, `SELECT id,vmid,state,upid,disk_slot,disk_serial,disk_bytes FROM extractor_allocations WHERE attempt_id=$1`, a.ID).Scan(&x.ID, &x.VMID, &x.State, &x.UPID, &x.DiskSlot, &x.Serial, &x.DiskBytes)
|
||||
return x, err
|
||||
}
|
||||
func (e *engine) extractLease(ctx context.Context, a attempt) error {
|
||||
var valid bool
|
||||
err := e.db.QueryRow(ctx, `SELECT lease_owner=$2 AND lease_until>now() FROM attempts WHERE id=$1`, a.ID, e.id).Scan(&valid)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !valid {
|
||||
return errors.New("extractor lease lost")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (e *engine) extractState(ctx context.Context, a attempt, x extractionAllocation, state, upid string) error {
|
||||
tag, err := e.db.Exec(ctx, `UPDATE extractor_allocations SET state=$3,upid=NULLIF($4,'') WHERE id=$1 AND EXISTS(SELECT 1 FROM attempts WHERE id=$2 AND lease_owner=$5 AND lease_until>now())`, x.ID, a.ID, state, upid, e.id)
|
||||
if err == nil && tag.RowsAffected() != 1 {
|
||||
return errors.New("extractor lease lost")
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (e *engine) extractHeld(parent context.Context, a attempt, l allocation, b Binding, cs clients, dir string) error {
|
||||
conf := e.cfg.Extractor
|
||||
if conf == nil {
|
||||
return errors.New("isolated read-only extractor not configured; all original disks retained")
|
||||
}
|
||||
timeout := conf.TimeoutSeconds
|
||||
if timeout < 30 || timeout > 300 {
|
||||
return errors.New("extractor timeout must be 30..300 seconds")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(parent, time.Duration(timeout)*time.Second)
|
||||
defer cancel()
|
||||
if conf.Node != b.Node || conf.SourceVMID < 100 || conf.SourceVMID == 7000 || conf.SourceVMID == 7001 || conf.MaxDiskBytes <= 0 {
|
||||
return errors.New("invalid same-node extractor source")
|
||||
}
|
||||
slot := conf.SourceDisk
|
||||
if slot == "" {
|
||||
slot = "scsi0"
|
||||
}
|
||||
if !diskSlotRE.MatchString(slot) {
|
||||
return errors.New("extractor requires explicit SCSI evidence slot")
|
||||
}
|
||||
var proof extractionProof
|
||||
raw, err := os.ReadFile(conf.ProofFile)
|
||||
if err != nil || len(raw) > 64<<10 || json.Unmarshal(raw, &proof) != nil || !proof.Passed || !proof.ReadOnlyVerified || !proof.NoEgress || proof.Node != conf.Node || proof.SourceVMID != conf.SourceVMID || proof.ConfigDigest != conf.ConfigDigest || !proof.ExpiresAt.After(time.Now()) || !shaRE.MatchString(proof.EvidenceSHA256) {
|
||||
return errors.New("qualified extractor read-only/no-egress evidence missing or expired")
|
||||
}
|
||||
if err = e.extractLease(ctx, a); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.CheckOwned(ctx, e.own(a, l, b)); err != nil {
|
||||
return err
|
||||
}
|
||||
state, err := cs.provisioner.Status(ctx, b.Node, l.VMID)
|
||||
if err != nil || state != "stopped" {
|
||||
return errors.New("evidence source must be confirmed stopped")
|
||||
}
|
||||
if _, err = e.extraction(ctx, a); err == nil {
|
||||
return errors.New("extractor allocation already recorded; reconcile without repeating extraction")
|
||||
}
|
||||
if !isNoRows(err) {
|
||||
return err
|
||||
}
|
||||
source, err := cs.provisioner.Config(ctx, b.Node, conf.SourceVMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
state, err = cs.provisioner.Status(ctx, b.Node, conf.SourceVMID)
|
||||
if err != nil || state != "stopped" || pve.Text(source["template"]) == "1" || pve.Text(source["digest"]) != conf.ConfigDigest {
|
||||
return errors.New("extractor source must be unchanged stopped ordinary VM")
|
||||
}
|
||||
if err = cs.provisioner.NoPending(ctx, b.Node, conf.SourceVMID, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
for k := range source {
|
||||
if strings.HasPrefix(k, "net") || strings.HasPrefix(k, "hostpci") || strings.HasPrefix(k, "usb") {
|
||||
return errors.New("extractor source must have no network or passthrough")
|
||||
}
|
||||
}
|
||||
if _, exists := source["scsi1"]; exists {
|
||||
return errors.New("extractor source scsi1 must be unused")
|
||||
}
|
||||
cfg, err := cs.provisioner.Config(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
disk := pve.Text(cfg[slot])
|
||||
if !ownVolume(disk, b.DiskStorage, l.VMID) || strings.Contains(disk, "media=cdrom") {
|
||||
return errors.New("evidence disk ownership/storage mismatch")
|
||||
}
|
||||
size, err := diskSize(disk)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = storageHeadroom(ctx, cs.provisioner, b, conf.MaxDiskBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
conn, err := sourceLock(ctx, e.db, "nextid-"+b.Endpoint)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer unlockSource(conn, "nextid-"+b.Endpoint)
|
||||
var next string
|
||||
if err = cs.provisioner.Do(ctx, "GET", "/cluster/nextid", nil, &next); err != nil {
|
||||
return err
|
||||
}
|
||||
id, err := strconv.Atoi(next)
|
||||
if err != nil {
|
||||
return errors.New("invalid extractor nextid")
|
||||
}
|
||||
for _, p := range e.cfg.protected() {
|
||||
if id == p {
|
||||
return errors.New("PVE suggested protected extractor VMID")
|
||||
}
|
||||
}
|
||||
var reserved bool
|
||||
if err = e.db.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM allocations WHERE vmid=$1 AND state<>'deleted') OR EXISTS(SELECT 1 FROM extractor_allocations WHERE vmid=$1 AND state<>'deleted')`, id).Scan(&reserved); err != nil {
|
||||
return err
|
||||
}
|
||||
if reserved {
|
||||
return errors.New("extractor VMID reserved by existing allocation")
|
||||
}
|
||||
x := extractionAllocation{ID: newID(), VMID: id, State: "reserved", DiskSlot: slot, Serial: "otche" + strings.ReplaceAll(a.ID, "-", "")[:16], DiskBytes: size}
|
||||
_, err = e.db.Exec(ctx, `INSERT INTO extractor_allocations(id,attempt_id,owner_id,node,vmid,state,disk_slot,disk_serial,disk_bytes,metadata)VALUES($1,$2,$3,$4,$5,'clone_intent',$6,$7,$8,jsonb_build_object('reserved_disk_bytes',$9::bigint,'source_vmid',$10::integer,'original_volume',$11::text))`, x.ID, a.ID, a.OwnerID, b.Node, id, slot, x.Serial, size, conf.MaxDiskBytes, conf.SourceVMID, strings.SplitN(disk, ",", 2)[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var upid string
|
||||
if err = cs.provisioner.Do(ctx, "POST", pve.VMPath(b.Node, conf.SourceVMID)+"/clone", url.Values{"newid": {strconv.Itoa(id)}, "full": {"1"}, "pool": {b.Pool}, "storage": {b.DiskStorage}, "name": {"otche-extractor-" + a.ID}}, &upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "cloning", upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
// Adoption requires the recorded successful clone task and exact name/pool.
|
||||
ecfg, err := cs.provisioner.Config(ctx, b.Node, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if pve.Text(ecfg["name"]) != "otche-extractor-"+a.ID {
|
||||
return errors.New("extractor clone name mismatch")
|
||||
}
|
||||
var pool struct {
|
||||
Members []struct {
|
||||
VMID int `json:"vmid"`
|
||||
Node string `json:"node"`
|
||||
Type string `json:"type"`
|
||||
} `json:"members"`
|
||||
}
|
||||
if err = cs.provisioner.Do(ctx, "GET", "/pools/"+b.Pool, nil, &pool); err != nil {
|
||||
return err
|
||||
}
|
||||
member := false
|
||||
for _, m := range pool.Members {
|
||||
if m.VMID == id && m.Node == b.Node && m.Type == "qemu" {
|
||||
member = true
|
||||
}
|
||||
}
|
||||
if !member {
|
||||
return errors.New("extractor clone pool mismatch")
|
||||
}
|
||||
for k, v := range ecfg {
|
||||
if strings.HasPrefix(k, "net") || strings.HasPrefix(k, "hostpci") || strings.HasPrefix(k, "usb") {
|
||||
return errors.New("extractor clone not isolated")
|
||||
}
|
||||
if diskSlotRE.MatchString(k) && !ownVolume(pve.Text(v), b.DiskStorage, id) {
|
||||
return errors.New("extractor boot disk ownership mismatch")
|
||||
}
|
||||
}
|
||||
o := e.extractOwn(a, x, b)
|
||||
if err = cs.provisioner.Do(ctx, "POST", pve.VMPath(b.Node, id)+"/config", url.Values{"tags": {o.Tags()}, "onboot": {"0"}}, &upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.CheckOwned(ctx, o); err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() {
|
||||
stopCtx, stopCancel := context.WithTimeout(context.Background(), 45*time.Second)
|
||||
defer stopCancel()
|
||||
current, lookupErr := e.extraction(stopCtx, a)
|
||||
if lookupErr != nil || current.State == "deleted" || e.extractLease(stopCtx, a) != nil {
|
||||
return
|
||||
}
|
||||
if cs.provisioner.CheckOwned(stopCtx, o) != nil {
|
||||
return
|
||||
}
|
||||
status, statusErr := cs.provisioner.Status(stopCtx, b.Node, x.VMID)
|
||||
if statusErr != nil || status == "stopped" {
|
||||
return
|
||||
}
|
||||
task, stopErr := cs.provisioner.OwnedTask(stopCtx, o, "POST", "/status/stop", nil)
|
||||
if stopErr == nil {
|
||||
stopErr = cs.provisioner.Task(stopCtx, b.Node, task)
|
||||
}
|
||||
if stopErr != nil {
|
||||
e.event(stopCtx, a, "retention", "Extractor external stop not confirmed; journal retained for watchdog/operator")
|
||||
}
|
||||
}()
|
||||
disk = diskOption(diskOption(disk, "ro", "1"), "serial", x.Serial)
|
||||
upid, err = cs.provisioner.OwnedTask(ctx, e.own(a, l, b), "POST", "/config", url.Values{slot: {disk}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
cfg, err = cs.provisioner.Config(ctx, b.Node, l.VMID)
|
||||
if err != nil || !diskHas(pve.Text(cfg[slot]), "ro", "1") {
|
||||
return errors.New("source evidence read-only flag not established")
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "move_intent", ""); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.CheckOwned(ctx, o); err != nil {
|
||||
return err
|
||||
}
|
||||
upid, err = cs.provisioner.OwnedTask(ctx, e.own(a, l, b), "POST", "/move_disk", url.Values{"disk": {slot}, "target-vmid": {strconv.Itoa(id)}, "target-disk": {"scsi1"}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "moving", upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
ecfg, err = cs.provisioner.Config(ctx, b.Node, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
evidence := pve.Text(ecfg["scsi1"])
|
||||
if !ownVolume(evidence, b.DiskStorage, id) || !diskHas(evidence, "ro", "1") || !diskHas(evidence, "serial", x.Serial) {
|
||||
return errors.New("reassigned evidence is not exact owned read-only disk")
|
||||
}
|
||||
if err = cs.provisioner.NoPending(ctx, b.Node, id, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "attached", ""); err != nil {
|
||||
return err
|
||||
}
|
||||
// Any later error leaves both VMs and the read-only evidence journal intact.
|
||||
upid, err = cs.provisioner.OwnedTask(ctx, o, "POST", "/status/start", nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "booting", upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
control := map[string]any{"command_id": a.CommandID, "serial": x.Serial, "disk_bytes": size, "max_bytes": e.cfg.MaxArtifactBytes, "include_crash_dump": conf.IncludeCrashDump}
|
||||
raw, _ = json.Marshal(control)
|
||||
controlPath := "/var/lib/otche/control/" + a.CommandID + ".json"
|
||||
ticker := time.NewTicker(2 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
err = cs.runtime.WriteControl(ctx, b.Node, id, controlPath, raw)
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "dispatch_intent", ""); err != nil {
|
||||
return err
|
||||
}
|
||||
pid, err := cs.runtime.Exec(ctx, b.Node, id, []string{"/usr/local/sbin/otche-extract", controlPath})
|
||||
if err != nil {
|
||||
return errors.New("extractor command acceptance unknown; do not repeat")
|
||||
}
|
||||
if _, err = cs.runtime.ExecWait(ctx, b.Node, id, pid); err != nil {
|
||||
return err
|
||||
}
|
||||
var manifest struct {
|
||||
CommandID string `json:"command_id"`
|
||||
Complete bool `json:"complete"`
|
||||
Errors []string `json:"errors"`
|
||||
Files []struct {
|
||||
Name string `json:"name"`
|
||||
Size int64 `json:"size"`
|
||||
} `json:"files"`
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
resultPath := "/var/lib/otche/export/" + a.CommandID + "/"
|
||||
if _, err = cs.runtime.ReadFile(ctx, b.Node, id, resultPath+"manifest.json", &buf, 1<<20); err != nil {
|
||||
return err
|
||||
}
|
||||
if json.Unmarshal(buf.Bytes(), &manifest) != nil || manifest.CommandID != a.CommandID || len(manifest.Files) > 64 {
|
||||
return errors.New("invalid extractor manifest")
|
||||
}
|
||||
total := int64(0)
|
||||
for _, f := range manifest.Files {
|
||||
if !regexp.MustCompile(`^[0-9]{3}-[A-Za-z0-9_.-]{1,120}$`).MatchString(f.Name) || f.Size < 0 || total+f.Size > e.cfg.MaxArtifactBytes {
|
||||
return errors.New("extractor export exceeds allowlist or size budget")
|
||||
}
|
||||
total += f.Size
|
||||
path := filepath.Join(dir, "extracted-"+f.Name)
|
||||
out, openErr := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
|
||||
if openErr != nil {
|
||||
return openErr
|
||||
}
|
||||
n, readErr := cs.runtime.ReadFile(ctx, b.Node, id, resultPath+f.Name, out, max(f.Size, 1))
|
||||
syncErr := out.Sync()
|
||||
out.Close()
|
||||
if readErr != nil || syncErr != nil || n != f.Size {
|
||||
return errors.New("bounded extractor transfer incomplete")
|
||||
}
|
||||
if err = e.publishFile(ctx, a, "extracted", filepath.Base(path), "application/octet-stream", path); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
mpath := filepath.Join(dir, "extraction-manifest.json")
|
||||
if err = atomicJSON(mpath, manifest); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.publishFile(ctx, a, "extraction_manifest", "extraction-manifest.json", "application/json", mpath); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.reconcileExtractor(ctx, a, l, b, cs); err != nil {
|
||||
return err
|
||||
}
|
||||
if !manifest.Complete {
|
||||
return fmt.Errorf("extractor retained original evidence: %d collection failures", len(manifest.Errors))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// reconcileExtractor stops the journaled isolated VM, returns borrowed evidence
|
||||
// read-only to the stopped Windows clone, and only then destroys extractor boot disks.
|
||||
// Unknown clone/move acceptance is retained rather than adopted or blindly replayed.
|
||||
func (e *engine) reconcileExtractor(ctx context.Context, a attempt, l allocation, b Binding, cs clients) error {
|
||||
x, err := e.extraction(ctx, a)
|
||||
if isNoRows(err) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if x.State == "deleted" {
|
||||
return nil
|
||||
}
|
||||
if err = e.extractLease(ctx, a); err != nil {
|
||||
return err
|
||||
}
|
||||
if x.UPID != nil {
|
||||
if err = cs.provisioner.Task(ctx, b.Node, *x.UPID); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
o := e.extractOwn(a, x, b)
|
||||
if err = cs.provisioner.CheckOwned(ctx, o); err != nil {
|
||||
return errors.New("extractor ownership not confirmed; retained journal requires operator reconciliation")
|
||||
}
|
||||
state, err := cs.provisioner.Status(ctx, b.Node, x.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if state != "stopped" {
|
||||
upid, err := cs.provisioner.OwnedTask(ctx, o, "POST", "/status/stop", nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "stopping", upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
state, err = cs.provisioner.Status(ctx, b.Node, x.VMID)
|
||||
if err != nil || state != "stopped" {
|
||||
return errors.New("extractor stop not confirmed")
|
||||
}
|
||||
if err = cs.provisioner.CheckOwned(ctx, e.own(a, l, b)); err != nil {
|
||||
return err
|
||||
}
|
||||
state, err = cs.provisioner.Status(ctx, b.Node, l.VMID)
|
||||
if err != nil || state != "stopped" {
|
||||
return errors.New("evidence destination must remain stopped")
|
||||
}
|
||||
ecfg, err := cs.provisioner.Config(ctx, b.Node, x.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cfg, err := cs.provisioner.Config(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if evidence := pve.Text(ecfg["scsi1"]); evidence != "" {
|
||||
if pve.Text(cfg[x.DiskSlot]) != "" || !ownVolume(evidence, b.DiskStorage, x.VMID) || !diskHas(evidence, "ro", "1") || !diskHas(evidence, "serial", x.Serial) {
|
||||
return errors.New("cannot safely return extractor evidence")
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "return_intent", ""); err != nil {
|
||||
return err
|
||||
}
|
||||
upid, err := cs.provisioner.OwnedTask(ctx, o, "POST", "/move_disk", url.Values{"disk": {"scsi1"}, "target-vmid": {strconv.Itoa(l.VMID)}, "target-disk": {x.DiskSlot}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "returning", upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
cfg, err = cs.provisioner.Config(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
evidence := pve.Text(cfg[x.DiskSlot])
|
||||
if !ownVolume(evidence, b.DiskStorage, l.VMID) || !diskHas(evidence, "ro", "1") || !diskHas(evidence, "serial", x.Serial) {
|
||||
return errors.New("returned evidence ownership/read-only verification failed")
|
||||
}
|
||||
ecfg, err = cs.provisioner.Config(ctx, b.Node, x.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if pve.Text(ecfg["scsi1"]) != "" {
|
||||
return errors.New("extractor still references evidence; deletion prohibited")
|
||||
}
|
||||
for key := range ecfg {
|
||||
if strings.HasPrefix(key, "unused") {
|
||||
return errors.New("extractor has unknown unused disk; deletion prohibited")
|
||||
}
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "delete_intent", ""); err != nil {
|
||||
return err
|
||||
}
|
||||
upid, err := cs.provisioner.OwnedTask(ctx, o, "DELETE", "", url.Values{"purge": {"0"}, "destroy-unreferenced-disks": {"0"}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.extractState(ctx, a, x, "deleting", upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
absent, err := vmAbsent(ctx, cs.provisioner, b.Node, x.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !absent {
|
||||
return errors.New("extractor deletion not confirmed")
|
||||
}
|
||||
return e.extractState(ctx, a, x, "deleted", "")
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
package worker
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestReadOnlyEvidenceIdentityBoundaries(t *testing.T) {
|
||||
source := "private:vm-8012-disk-0,discard=on,size=64G,ro=0"
|
||||
protected := diskOption(diskOption(source, "ro", "1"), "serial", "otche0123456789abcdef")
|
||||
if !diskHas(protected, "ro", "1") || diskHas(protected, "ro", "0") {
|
||||
t.Fatal("old writable flag survived read-only preparation")
|
||||
}
|
||||
if !ownVolume(protected, "private", 8012) || ownVolume(protected, "private", 801) || ownVolume(protected, "other", 8012) {
|
||||
t.Fatal("evidence identity accepted foreign storage or VM prefix")
|
||||
}
|
||||
if n, err := diskSize(protected); err != nil || n != 64<<30 {
|
||||
t.Fatalf("allocated size mismatch: %d %v", n, err)
|
||||
}
|
||||
for _, invalid := range []string{"private:vm-8012-disk-0", "private:vm-8012-disk-0,size=-1G", "private:vm-8012-disk-0,size=999999999999999999T"} {
|
||||
if _, err := diskSize(invalid); err == nil {
|
||||
t.Fatalf("invalid disk size accepted: %q", invalid)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDirectoryEvidenceVolumeOwnership(t *testing.T) {
|
||||
for _, format := range []string{"qcow2", "raw", "vmdk"} {
|
||||
volume := "private:8012/vm-8012-disk-0." + format + ",size=64G,ro=1"
|
||||
if !ownVolume(volume, "private", 8012) {
|
||||
t.Fatalf("owned directory disk rejected: %q", volume)
|
||||
}
|
||||
}
|
||||
for _, volume := range []string{
|
||||
"private:8013/vm-8012-disk-0.qcow2",
|
||||
"private:8012/vm-8013-disk-0.qcow2",
|
||||
"private:8012/../8013/vm-8012-disk-0.qcow2",
|
||||
"private:8012/vm-8012-disk-0.qcow2/other",
|
||||
"private:8012/base-8012-disk-0.qcow2",
|
||||
"other:8012/vm-8012-disk-0.qcow2",
|
||||
"private:8012/vm-8012-disk-0.iso",
|
||||
"private:vm-8012-disk-0/../vm-8013-disk-0",
|
||||
} {
|
||||
if ownVolume(volume, "private", 8012) {
|
||||
t.Fatalf("foreign or malformed directory disk accepted: %q", volume)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCDMediaIdentityAllowsPVEAnnotationsButRejectsAmbiguity(t *testing.T) {
|
||||
const volume = "private:iso/otche-job.iso"
|
||||
for _, tc := range []struct {
|
||||
name, value, want string
|
||||
matches bool
|
||||
}{
|
||||
{"inserted_with_size", volume + ",media=cdrom,size=370K", volume, true},
|
||||
{"empty_with_reordered_options", "none,format=raw,size=0,media=cdrom", "none", true},
|
||||
{"wrong_volume", "private:iso/other.iso,media=cdrom", volume, false},
|
||||
{"inserted_not_empty", volume + ",media=cdrom", "none", false},
|
||||
{"non_cd", volume + ",media=disk", volume, false},
|
||||
{"missing_media", volume + ",size=370K", volume, false},
|
||||
{"duplicate_media", volume + ",media=cdrom,media=cdrom", volume, false},
|
||||
{"conflicting_media", volume + ",media=cdrom,media=disk", volume, false},
|
||||
{"bare_second_volume", volume + ",media=cdrom,private:iso/other.iso", volume, false},
|
||||
{"file_alias", volume + ",media=cdrom,file=" + volume, volume, false},
|
||||
{"volume_alias", volume + ",media=cdrom,volume=private:iso/other.iso", volume, false},
|
||||
{"media_prefix", volume + ",media=cdrom2", volume, false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := cdromMatches(tc.value, tc.want); got != tc.matches {
|
||||
t.Fatalf("CD identity match=%v for %q, expected volume %q", got, tc.value, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,221 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"otche/internal/grub"
|
||||
)
|
||||
|
||||
// buildGrubArchive trusts neither guest metadata nor archive member names. The
|
||||
// caller supplies a bounded guest transport, not a local path supplied by a job.
|
||||
func buildGrubArchive(dst io.Writer, requested []string, raw json.RawMessage, limit int64, fetch func(int, io.Writer, int64) (int64, error)) (grub.Report, error) {
|
||||
var guest struct {
|
||||
Files []grub.File `json:"files"`
|
||||
}
|
||||
result := grub.Report{Requested: len(requested), Files: make([]grub.File, 0, len(requested))}
|
||||
if err := grub.Validate(requested); err != nil {
|
||||
return result, err
|
||||
}
|
||||
if json.Unmarshal(raw, &guest) != nil || len(guest.Files) != len(requested) {
|
||||
return result, errors.New("Grub result missing or mismatched; evidence retained")
|
||||
}
|
||||
z := zip.NewWriter(dst)
|
||||
var total int64
|
||||
complete := true
|
||||
for i, v := range guest.Files {
|
||||
if v.RequestedPath != requested[i] || len(v.Error) > 2048 || (v.ResolvedPath != nil && grub.Validate([]string{*v.ResolvedPath}) != nil) {
|
||||
return result, errors.New("Grub path mapping is invalid")
|
||||
}
|
||||
v.Member = nil
|
||||
if v.Status != "collected" {
|
||||
complete = false
|
||||
}
|
||||
if v.Size != nil {
|
||||
if (v.Status != "collected" && v.Status != "changed") || *v.Size < 0 || *v.Size > min(int64(8<<20), limit) || *v.Size > limit-total || v.SHA256 == nil || !shaRE.MatchString(*v.SHA256) || v.ResolvedPath == nil || v.Snapshot == nil || v.Snapshot.OpenSize < *v.Size || v.Snapshot.OpenSize > min(int64(8<<20), limit) || v.Snapshot.Consistency != "best_effort" || v.Snapshot.Changed != (v.Status == "changed") || (v.Status == "collected" && v.Error != "") || (v.Status == "changed" && v.Error == "") {
|
||||
return result, errors.New("Grub collected metadata exceeds bounds")
|
||||
}
|
||||
if v.Status == "collected" && v.Snapshot.OpenSize != *v.Size {
|
||||
return result, errors.New("truncated Grub snapshot must be explicitly changed")
|
||||
}
|
||||
member := grub.Member(i, requested[i])
|
||||
v.Member = &member
|
||||
w, err := z.CreateHeader(&zip.FileHeader{Name: member, Method: zip.Store})
|
||||
if err != nil {
|
||||
return result, err
|
||||
}
|
||||
hash := sha256.New()
|
||||
n, err := fetch(i, io.MultiWriter(w, hash), max(*v.Size, 1))
|
||||
if err != nil || n != *v.Size || hex.EncodeToString(hash.Sum(nil)) != *v.SHA256 {
|
||||
return result, errors.New("Grub transport size/hash mismatch; evidence retained")
|
||||
}
|
||||
total += n
|
||||
result.Collected++
|
||||
} else {
|
||||
switch v.Status {
|
||||
case "missing", "access_denied", "invalid_path", "changed", "oversize", "error":
|
||||
if v.Error == "" || v.SHA256 != nil || v.Snapshot != nil {
|
||||
return result, errors.New("Grub failure metadata is invalid")
|
||||
}
|
||||
default:
|
||||
return result, errors.New("Grub file status is invalid")
|
||||
}
|
||||
}
|
||||
result.Files = append(result.Files, v)
|
||||
}
|
||||
result.Status = "partial"
|
||||
if complete {
|
||||
result.Status = "complete"
|
||||
} else if result.Collected == 0 {
|
||||
result.Status = "empty"
|
||||
}
|
||||
w, err := z.CreateHeader(&zip.FileHeader{Name: "manifest.json", Method: zip.Store})
|
||||
if err != nil {
|
||||
return result, err
|
||||
}
|
||||
if err = json.NewEncoder(w).Encode(result); err != nil {
|
||||
return result, err
|
||||
}
|
||||
return result, z.Close()
|
||||
}
|
||||
|
||||
func (e *engine) collectGrub(ctx context.Context, a attempt, l allocation, b Binding, cs clients, requested []string, report json.RawMessage, dir string) (json.RawMessage, error) {
|
||||
if err := cs.provisioner.CheckOwned(ctx, e.own(a, l, b)); err != nil {
|
||||
return report, err
|
||||
}
|
||||
var fields map[string]json.RawMessage
|
||||
if json.Unmarshal(report, &fields) != nil {
|
||||
return report, errors.New("Grub report is unavailable")
|
||||
}
|
||||
prefix := `C:\ProgramData\Otche\control\grub-` + a.CommandID
|
||||
for {
|
||||
var exported struct {
|
||||
CommandID string `json:"command_id"`
|
||||
Ready bool `json:"ready"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
if err := readGuestJSON(ctx, cs.runtime, b, l, prefix+".json", &exported); err == nil {
|
||||
if exported.CommandID != a.CommandID || !exported.Ready {
|
||||
return report, errors.New("Grub protected export failed; original evidence retained")
|
||||
}
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return report, errors.New("Grub export unavailable before deadline; evidence retained")
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
f, err := os.CreateTemp(dir, "grub-*.zip")
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
target := f.Name()
|
||||
publishAttempted := false
|
||||
defer func() {
|
||||
if !publishAttempted {
|
||||
os.Remove(target)
|
||||
}
|
||||
}()
|
||||
archiveHash := sha256.New()
|
||||
summary, err := buildGrubArchive(io.MultiWriter(f, archiveHash), requested, fields["grub"], min(int64(32<<20), e.cfg.MaxArtifactBytes/2), func(index int, dst io.Writer, limit int64) (int64, error) {
|
||||
return cs.runtime.ReadFile(ctx, b.Node, l.VMID, prefix+fmt.Sprintf("-%03d.bin", index+1), dst, limit)
|
||||
})
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return report, err
|
||||
}
|
||||
info, statErr := f.Stat()
|
||||
syncErr := f.Sync()
|
||||
closeErr := f.Close()
|
||||
if statErr != nil {
|
||||
return report, statErr
|
||||
}
|
||||
if syncErr != nil {
|
||||
return report, syncErr
|
||||
}
|
||||
if closeErr != nil {
|
||||
return report, closeErr
|
||||
}
|
||||
if runtime.GOOS != "windows" {
|
||||
directory, err := os.Open(dir)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
syncErr := directory.Sync()
|
||||
closeErr := directory.Close()
|
||||
if syncErr != nil {
|
||||
return report, syncErr
|
||||
}
|
||||
if closeErr != nil {
|
||||
return report, closeErr
|
||||
}
|
||||
}
|
||||
fields["grub"], err = json.Marshal(summary)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
updated, err := json.Marshal(fields)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
key, err := filepath.Rel(e.root, target)
|
||||
if err != nil || filepath.IsAbs(key) || strings.HasPrefix(key, "..") {
|
||||
return report, errors.New("Grub archive escaped private root")
|
||||
}
|
||||
// Serialize publication with claim changes; archive and report become visible
|
||||
// atomically, and a stale generation can neither publish nor overwrite evidence.
|
||||
tx, err := e.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
var lease bool
|
||||
if err = tx.QueryRow(ctx, "SELECT COALESCE(lease_owner=$2 AND lease_until>now(),false) FROM attempts WHERE id=$1 FOR UPDATE", a.ID, e.id).Scan(&lease); err != nil {
|
||||
return report, err
|
||||
}
|
||||
if !lease {
|
||||
return report, errors.New("Grub publication lease lost")
|
||||
}
|
||||
var exists bool
|
||||
if err = tx.QueryRow(ctx, "SELECT EXISTS(SELECT 1 FROM artifacts WHERE attempt_id=$1 AND kind='grub_archive')", a.ID).Scan(&exists); err != nil {
|
||||
return report, err
|
||||
}
|
||||
if exists {
|
||||
return report, errors.New("immutable Grub archive already published; refusing replacement")
|
||||
}
|
||||
if _, err = tx.Exec(ctx, "INSERT INTO artifacts(id,job_id,attempt_id,kind,filename,content_type,size,sha256,storage_key) VALUES($1,$2,$3,'grub_archive','grub.zip','application/zip',$4,$5,$6)", newID(), a.JobID, a.ID, info.Size(), hex.EncodeToString(archiveHash.Sum(nil)), filepath.ToSlash(key)); err != nil {
|
||||
return report, err
|
||||
}
|
||||
tag, err := tx.Exec(ctx, "UPDATE attempts SET report=$2 WHERE id=$1 AND lease_owner=$3 AND lease_until>now()", a.ID, updated, e.id)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
if tag.RowsAffected() != 1 {
|
||||
return report, errors.New("Grub publication lease expired")
|
||||
}
|
||||
publishAttempted = true // A lost commit response must not unlink possibly published bytes.
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return report, err
|
||||
}
|
||||
return updated, nil
|
||||
}
|
||||
|
||||
func grubReservation(a attempt, limit int64) int64 {
|
||||
var opts settings
|
||||
if json.Unmarshal(a.Settings, &opts) == nil && len(opts.GrubPaths) > 0 {
|
||||
return limit
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"otche/internal/grub"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGrubArchiveSnapshotsAndMissing(t *testing.T) {
|
||||
paths := []string{`C:\a\same.log`, `C:\b\same.log`, `C:\empty.log`, `C:\missing.log`}
|
||||
contents := [][]byte{[]byte("one\x00"), []byte("two"), {}, nil}
|
||||
files := make([]grub.File, 4)
|
||||
for i := 0; i < 3; i++ {
|
||||
size := int64(len(contents[i]))
|
||||
sum := sha256.Sum256(contents[i])
|
||||
hash := hex.EncodeToString(sum[:])
|
||||
path := paths[i]
|
||||
files[i] = grub.File{RequestedPath: path, ResolvedPath: &path, Status: "collected", Size: &size, SHA256: &hash, Snapshot: &grub.Snapshot{OpenSize: size, Consistency: "best_effort"}}
|
||||
}
|
||||
files[1].Status = "changed"
|
||||
files[1].Error = "File changed during snapshot"
|
||||
files[1].Snapshot.Changed = true
|
||||
files[3] = grub.File{RequestedPath: paths[3], Status: "missing", Error: "Not found"}
|
||||
raw, _ := json.Marshal(grub.Report{Files: files})
|
||||
var output bytes.Buffer
|
||||
report, err := buildGrubArchive(&output, paths, raw, 100, func(i int, w io.Writer, limit int64) (int64, error) { n, e := w.Write(contents[i]); return int64(n), e })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if report.Status != "partial" || report.Collected != 3 || report.Files[3].Size != nil || report.Files[2].Size == nil || *report.Files[2].Size != 0 {
|
||||
t.Fatalf("dishonest snapshot report: %+v", report)
|
||||
}
|
||||
archive, err := zip.NewReader(bytes.NewReader(output.Bytes()), int64(output.Len()))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 3; i++ {
|
||||
f, err := archive.Open(*report.Files[i].Member)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, err := io.ReadAll(f)
|
||||
f.Close()
|
||||
if err != nil || !bytes.Equal(data, contents[i]) {
|
||||
t.Fatalf("wrong archived bytes %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
if *report.Files[0].Member == *report.Files[1].Member {
|
||||
t.Fatal("same basenames overwritten")
|
||||
}
|
||||
f, err := archive.Open("manifest.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer f.Close()
|
||||
var stored grub.Report
|
||||
if err = json.NewDecoder(f).Decode(&stored); err != nil || stored.Files[1].Status != "changed" || stored.Files[3].Status != "missing" {
|
||||
t.Fatal("ZIP lacks honest per-file manifest")
|
||||
}
|
||||
for _, failure := range []string{"hash", "transport", "quota", "mapping", "false_complete"} {
|
||||
t.Run(failure, func(t *testing.T) {
|
||||
var dst bytes.Buffer
|
||||
limit := int64(100)
|
||||
copyFiles := append([]grub.File(nil), files...)
|
||||
if failure == "quota" {
|
||||
limit = 2
|
||||
}
|
||||
if failure == "mapping" {
|
||||
copyFiles[0].RequestedPath = `C:\other.log`
|
||||
}
|
||||
if failure == "false_complete" {
|
||||
snapshot := *copyFiles[0].Snapshot
|
||||
snapshot.OpenSize++
|
||||
copyFiles[0].Snapshot = &snapshot
|
||||
}
|
||||
r, _ := json.Marshal(grub.Report{Files: copyFiles})
|
||||
_, err := buildGrubArchive(&dst, paths, r, limit, func(i int, w io.Writer, l int64) (int64, error) {
|
||||
if failure == "transport" {
|
||||
return 0, errors.New("lost guest")
|
||||
}
|
||||
data := contents[i]
|
||||
if failure == "hash" {
|
||||
data = []byte("bad")
|
||||
}
|
||||
n, e := w.Write(data)
|
||||
return int64(n), e
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("critical failure allowed successful archive")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGrubAllMissingIsManifestOnly(t *testing.T) {
|
||||
paths := []string{`C:\missing.log`}
|
||||
raw, _ := json.Marshal(grub.Report{Files: []grub.File{{RequestedPath: paths[0], Status: "missing", Error: "File not found"}}})
|
||||
var dst bytes.Buffer
|
||||
report, err := buildGrubArchive(&dst, paths, raw, 100, func(int, io.Writer, int64) (int64, error) { t.Fatal("missing file fetched"); return 0, nil })
|
||||
if err != nil || report.Status != "empty" || report.Collected != 0 {
|
||||
t.Fatalf("empty archive result: %+v %v", report, err)
|
||||
}
|
||||
z, err := zip.NewReader(bytes.NewReader(dst.Bytes()), int64(dst.Len()))
|
||||
if err != nil || len(z.File) != 1 || z.File[0].Name != "manifest.json" {
|
||||
t.Fatal("missing paths must produce manifest-only ZIP")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"os"
|
||||
"otche/internal/store"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A genuine lease generation regression: old workers must not regain authority
|
||||
// merely because the same process reclaims an expired attempt.
|
||||
func TestLeaseGenerationFencesExpiredClaimAndAllowsCancelledCleanup(t *testing.T) {
|
||||
dsn := os.Getenv("OTCHE_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("OTCHE_TEST_DATABASE_URL required for isolated PostgreSQL regression")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
admin, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Close()
|
||||
schema := "test_" + strings.ReplaceAll(newID(), "-", "")
|
||||
if _, err = admin.Exec(ctx, "CREATE SCHEMA "+schema); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Exec(context.Background(), "DROP SCHEMA "+schema+" CASCADE")
|
||||
cfg, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg.ConnConfig.RuntimeParams["search_path"] = schema
|
||||
db, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if err = store.Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
owner, profile, revision, upload, job, run, aid, command := newID(), newID(), newID(), newID(), newID(), newID(), newID(), newID()
|
||||
statements := []struct {
|
||||
sql string
|
||||
args []any
|
||||
}{
|
||||
{"INSERT INTO users(id,username,password_hash,role)VALUES($1,'lease-owner','unused','operator')", []any{owner}},
|
||||
{"INSERT INTO profiles(id,name,os,architecture)VALUES($1,'lease-profile','Windows','x64')", []any{profile}},
|
||||
{"INSERT INTO revisions(id,profile_id,source_ref,fingerprint,config_digest)VALUES($1,$2,'source','fp','cfg')", []any{revision, profile}},
|
||||
{"INSERT INTO uploads(id,owner_id,filename,size,sha256,storage_key)VALUES($1,$2,'safe.exe',1,'hash','test-upload')", []any{upload, owner}},
|
||||
{"INSERT INTO jobs(id,owner_id,upload_id,execution_filename,settings)VALUES($1,$2,$3,'safe.exe','{}')", []any{job, owner, upload}},
|
||||
{"INSERT INTO runs(id,job_id,profile_id,revision_id)VALUES($1,$2,$3,$4)", []any{run, job, profile, revision}},
|
||||
{"INSERT INTO attempts(id,run_id,command_id)VALUES($1,$2,$3)", []any{aid, run, command}},
|
||||
}
|
||||
for _, s := range statements {
|
||||
if _, err = db.Exec(ctx, s.sql, s.args...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
e := engine{db: db, cfg: Config{Concurrent: 1}, id: "same-process"}
|
||||
first, err := e.claim(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := e
|
||||
old.id = first.Lease
|
||||
if err = old.phase(ctx, first, "provisioning"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = e.claim(ctx); !isNoRows(err) {
|
||||
t.Fatalf("active lease allowed second claim: %v", err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, "UPDATE attempts SET lease_until=now()-interval '1 second' WHERE id=$1", aid); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second, err := e.claim(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if first.Lease == second.Lease {
|
||||
t.Fatal("reclaimed attempt reused old lease generation")
|
||||
}
|
||||
current := e
|
||||
current.id = second.Lease
|
||||
if old.leaseAlive(ctx, first) == nil || old.phase(ctx, first, "running") == nil {
|
||||
t.Fatal("expired generation retained mutation authority")
|
||||
}
|
||||
if err = current.phase(ctx, second, "collecting"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, "UPDATE jobs SET cancel_requested=true WHERE id=$1", job); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if current.fence(ctx, second) == nil {
|
||||
t.Fatal("cancel failed to revoke execution permission")
|
||||
}
|
||||
if err = current.leaseAlive(ctx, second); err != nil {
|
||||
t.Fatalf("cancel must retain cleanup authority: %v", err)
|
||||
}
|
||||
var phase string
|
||||
if err = db.QueryRow(ctx, "SELECT phase FROM attempts WHERE id=$1", aid).Scan(&phase); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if phase != "collecting" {
|
||||
t.Fatalf("stale worker overwrote phase: %s", phase)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,259 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"otche/internal/pve"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func isoLabel(job string) string {
|
||||
return "OT" + strings.ToUpper(strings.ReplaceAll(job, "-", "")[:14])
|
||||
}
|
||||
|
||||
var errMediaAbsent = errors.New("uploaded job ISO missing")
|
||||
|
||||
func (e *engine) media(ctx context.Context, a attempt, b Binding, cs clients) (string, error) {
|
||||
conn, err := sourceLock(ctx, e.db, "media-"+a.JobID+"-"+b.Node)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer unlockSource(conn, "media-"+a.JobID+"-"+b.Node)
|
||||
filename := "otche-" + a.JobID + ".iso"
|
||||
volume := b.ISOStorage + ":iso/" + filename
|
||||
var state, existing string
|
||||
err = e.db.QueryRow(ctx, "SELECT state,volume FROM media WHERE job_id=$1 AND node=$2", a.JobID, b.Node).Scan(&state, &existing)
|
||||
if err == nil {
|
||||
if existing != volume {
|
||||
return "", errors.New("media binding differs from durable record")
|
||||
}
|
||||
if state == "ready" {
|
||||
return volume, e.mediaExists(ctx, cs.uploader, b, volume)
|
||||
}
|
||||
if state == "deleted" {
|
||||
return e.recreateDeletedMedia(ctx, a, b, cs, volume)
|
||||
}
|
||||
if strings.HasPrefix(state, "uploading:") {
|
||||
taskCtx, cancel := context.WithTimeout(ctx, time.Duration(e.cfg.TaskTimeoutSeconds)*time.Second)
|
||||
defer cancel()
|
||||
if err = cs.uploader.Task(taskCtx, b.Node, strings.TrimPrefix(state, "uploading:")); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = e.mediaExists(ctx, cs.uploader, b, volume); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE media SET state='ready' WHERE job_id=$1 AND node=$2", a.JobID, b.Node)
|
||||
return volume, err
|
||||
}
|
||||
return "", errors.New("media upload/delete acceptance unknown; media is never blindly re-uploaded")
|
||||
}
|
||||
if !isNoRows(err) {
|
||||
return "", err
|
||||
}
|
||||
input, err := contained(e.root, a.StorageKey)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if filepath.Base(a.Filename) != a.Filename || strings.ContainsAny(a.Filename, "/\\\x00:") || a.Filename == "." || a.Filename == ".." {
|
||||
return "", errors.New("invalid immutable execution filename")
|
||||
}
|
||||
src, err := os.Open(input)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
h := sha256.New()
|
||||
size, err := io.Copy(h, src)
|
||||
src.Close()
|
||||
if err != nil || size != a.Size || hex.EncodeToString(h.Sum(nil)) != a.SHA256 {
|
||||
return "", errors.New("original upload integrity failed")
|
||||
}
|
||||
base := filepath.Join(e.root, "media", a.JobID, b.Node)
|
||||
if err = os.MkdirAll(base, 0700); err != nil {
|
||||
return "", err
|
||||
}
|
||||
stage := filepath.Join(base, "tree")
|
||||
if err = os.MkdirAll(filepath.Join(stage, "sample"), 0700); err != nil {
|
||||
return "", err
|
||||
}
|
||||
sample := filepath.Join(stage, "sample", a.Filename)
|
||||
if _, err = os.Lstat(sample); os.IsNotExist(err) {
|
||||
if err = os.Link(input, sample); err != nil {
|
||||
return "", errors.New("ISO staging needs same-volume immutable upload hardlink")
|
||||
}
|
||||
} else if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = atomicJSON(filepath.Join(stage, "job.json"), map[string]any{"job_id": a.JobID, "sha256": a.SHA256, "filename": a.Filename, "iso_label": isoLabel(a.JobID)}); err != nil {
|
||||
return "", err
|
||||
}
|
||||
isoPath := filepath.Join(base, filename)
|
||||
iso, err := os.OpenFile(isoPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return "", errors.New("ISO generation artifact already exists or unavailable; inspect before retry")
|
||||
}
|
||||
buildCtx, cancel := context.WithTimeout(ctx, time.Duration(e.cfg.TaskTimeoutSeconds)*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(buildCtx, e.cfg.Xorriso, "-as", "mkisofs", "-quiet", "-iso-level", "3", "-J", "-joliet-long", "-R", "-V", isoLabel(a.JobID), "-o", "-", stage)
|
||||
hash := sha256.New()
|
||||
cmd.Stdout = io.MultiWriter(iso, hash)
|
||||
cmd.Stderr = io.Discard
|
||||
err = cmd.Run()
|
||||
syncErr := iso.Sync()
|
||||
closeErr := iso.Close()
|
||||
if err != nil || syncErr != nil || closeErr != nil {
|
||||
return "", errors.New("bounded xorriso ISO generation failed")
|
||||
}
|
||||
info, err := os.Stat(isoPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
isoSHA := hex.EncodeToString(hash.Sum(nil))
|
||||
if err = atomicJSON(filepath.Join(base, "iso-manifest.json"), map[string]any{"sha256": isoSHA, "size": info.Size(), "original_sha256": a.SHA256, "volume": volume}); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = storageHeadroom(ctx, cs.provisioner, b, info.Size()); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = e.fence(ctx, a); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "INSERT INTO media(id,job_id,owner_id,node,volume,state) VALUES($1,$2,$3,$4,$5,'upload_intent')", newID(), a.JobID, a.OwnerID, b.Node, volume)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
f, err := os.Open(isoPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
upid, err := cs.uploader.Upload(buildCtx, b.Node, b.ISOStorage, filename, isoSHA, f)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE media SET state=$3 WHERE job_id=$1 AND node=$2", a.JobID, b.Node, "uploading:"+upid)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = cs.uploader.Task(buildCtx, b.Node, upid); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = e.mediaExists(ctx, cs.uploader, b, volume); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE media SET state='ready' WHERE job_id=$1 AND node=$2", a.JobID, b.Node)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = e.publishFile(ctx, a, "media_manifest", "iso-manifest.json", "application/json", filepath.Join(base, "iso-manifest.json")); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return volume, nil
|
||||
}
|
||||
func (e *engine) mediaExists(ctx context.Context, c *pve.Client, b Binding, volume string) error {
|
||||
var items []struct {
|
||||
VolID string `json:"volid"`
|
||||
Size int64 `json:"size"`
|
||||
}
|
||||
if err := c.Do(ctx, "GET", "/nodes/"+b.Node+"/storage/"+b.ISOStorage+"/content", url.Values{"content": {"iso"}}, &items); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, v := range items {
|
||||
if v.VolID == volume && v.Size > 0 {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return errMediaAbsent
|
||||
}
|
||||
func (e *engine) removeMedia(ctx context.Context, a attempt, b Binding, cs clients) error {
|
||||
var active int
|
||||
if err := e.db.QueryRow(ctx, "SELECT count(*) FROM allocations l JOIN attempts a ON a.id=l.attempt_id JOIN runs r ON r.id=a.run_id WHERE r.job_id=$1 AND l.state<>'deleted'", a.JobID).Scan(&active); err != nil {
|
||||
return err
|
||||
}
|
||||
if active > 0 {
|
||||
return nil
|
||||
}
|
||||
var remaining int
|
||||
if err := e.db.QueryRow(ctx, "SELECT count(*) FROM attempts a JOIN runs r ON r.id=a.run_id WHERE r.job_id=$1 AND a.id<>$2 AND a.phase<>'finished'", a.JobID, a.ID).Scan(&remaining); err != nil {
|
||||
return err
|
||||
}
|
||||
if remaining > 0 {
|
||||
return nil
|
||||
}
|
||||
var volume, state string
|
||||
err := e.db.QueryRow(ctx, "SELECT volume,state FROM media WHERE job_id=$1 AND node=$2 AND owner_id=$3", a.JobID, b.Node, a.OwnerID).Scan(&volume, &state)
|
||||
if isNoRows(err) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if state == "deleted" {
|
||||
return nil
|
||||
}
|
||||
expected := b.ISOStorage + ":iso/otche-" + a.JobID + ".iso"
|
||||
if volume != expected {
|
||||
return errors.New("media ownership mismatch")
|
||||
}
|
||||
if state != "ready" && state != "delete_intent" {
|
||||
return errors.New("uncertain media retained for administrative inspection")
|
||||
}
|
||||
// Every live VM visible to the trusted provisioner is checked for references.
|
||||
var vms []struct {
|
||||
VMID int `json:"vmid"`
|
||||
}
|
||||
if err = cs.provisioner.Do(ctx, "GET", "/nodes/"+b.Node+"/qemu", nil, &vms); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, vm := range vms {
|
||||
cfg, err := cs.provisioner.Config(ctx, b.Node, vm.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, v := range cfg {
|
||||
if strings.Contains(pve.Text(v), volume) {
|
||||
return errors.New("media still referenced by a VM")
|
||||
}
|
||||
}
|
||||
}
|
||||
if err = e.leaseAlive(ctx, a); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE media SET state='delete_intent' WHERE job_id=$1 AND node=$2", a.JobID, b.Node)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var result json.RawMessage
|
||||
err = cs.housekeeping.Do(ctx, "DELETE", "/nodes/"+b.Node+"/storage/"+b.ISOStorage+"/content/"+url.PathEscape(volume), nil, &result)
|
||||
if err != nil && !pve.IsNotFound(err) {
|
||||
return err
|
||||
}
|
||||
if err == nil && string(result) != "null" {
|
||||
var upid string
|
||||
if json.Unmarshal(result, &upid) != nil {
|
||||
return errors.New("malformed media deletion task")
|
||||
}
|
||||
if err = cs.housekeeping.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err = e.mediaExists(ctx, cs.housekeeping, b, volume); !errors.Is(err, errMediaAbsent) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return errors.New("ISO remains after deletion")
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE media SET state='deleted' WHERE job_id=$1 AND node=$2", a.JobID, b.Node)
|
||||
if err == nil {
|
||||
e.event(ctx, a, "media_generation", "Job ISO generation deletion confirmed after all owned attachments released")
|
||||
}
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
// An explicit user retry may recreate a previously CONFIRMED deleted generation.
|
||||
// Unknown upload/delete intent never enters this path; bytes come from the same
|
||||
// retained ISO and are hashed against the original immutable ISO manifest.
|
||||
func (e *engine) recreateDeletedMedia(ctx context.Context, a attempt, b Binding, cs clients, volume string) (string, error) {
|
||||
base := filepath.Join(e.root, "media", a.JobID, b.Node)
|
||||
raw, err := os.ReadFile(filepath.Join(base, "iso-manifest.json"))
|
||||
if err != nil {
|
||||
return "", errors.New("deleted job media cannot be recreated without its immutable ISO manifest")
|
||||
}
|
||||
var m struct {
|
||||
SHA256 string `json:"sha256"`
|
||||
Size int64 `json:"size"`
|
||||
Original string `json:"original_sha256"`
|
||||
Volume string `json:"volume"`
|
||||
}
|
||||
if json.Unmarshal(raw, &m) != nil || m.Volume != volume || m.Original != a.SHA256 || !shaRE.MatchString(m.SHA256) {
|
||||
return "", errors.New("media generation manifest integrity mismatch")
|
||||
}
|
||||
filename := "otche-" + a.JobID + ".iso"
|
||||
f, err := os.Open(filepath.Join(base, filename))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(h, f)
|
||||
if err != nil || n != m.Size || hex.EncodeToString(h.Sum(nil)) != m.SHA256 {
|
||||
return "", errors.New("retained ISO changed; retry refused")
|
||||
}
|
||||
if _, err = f.Seek(0, 0); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = e.fence(ctx, a); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = storageHeadroom(ctx, cs.provisioner, b, m.Size); err != nil {
|
||||
return "", err
|
||||
}
|
||||
tag, err := e.db.Exec(ctx, "UPDATE media SET state='upload_intent' WHERE job_id=$1 AND node=$2 AND owner_id=$3 AND volume=$4 AND state='deleted'", a.JobID, b.Node, a.OwnerID, volume)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if tag.RowsAffected() != 1 {
|
||||
return "", errors.New("media generation changed concurrently")
|
||||
}
|
||||
e.event(ctx, a, "media_generation", "Explicit new attempt recreates previously confirmed deleted job ISO; original SHA-256 "+m.SHA256)
|
||||
bounded, cancel := context.WithTimeout(ctx, time.Duration(e.cfg.TaskTimeoutSeconds)*time.Second)
|
||||
defer cancel()
|
||||
upid, err := cs.uploader.Upload(bounded, b.Node, b.ISOStorage, filename, m.SHA256, f)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err = e.db.Exec(ctx, "UPDATE media SET state=$3 WHERE job_id=$1 AND node=$2", a.JobID, b.Node, "uploading:"+upid); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = cs.uploader.Task(bounded, b.Node, upid); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = e.mediaExists(ctx, cs.uploader, b, volume); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE media SET state='ready' WHERE job_id=$1 AND node=$2", a.JobID, b.Node)
|
||||
return volume, err
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
"unicode/utf16"
|
||||
)
|
||||
|
||||
func TestISOLabelSurvivesPrimaryAndJolietDescriptors(t *testing.T) {
|
||||
binaryPath, err := exec.LookPath("xorriso")
|
||||
if err != nil {
|
||||
t.Skip("requires real xorriso on PATH")
|
||||
}
|
||||
const job = "d248cefa-179c-4bea-a59f-9fa7ae36e841"
|
||||
label := isoLabel(job)
|
||||
stage := t.TempDir()
|
||||
if err = os.WriteFile(filepath.Join(stage, "job.json"), []byte(`{"job_id":"`+job+`"}`), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, binaryPath, "-as", "mkisofs", "-quiet", "-iso-level", "3", "-J", "-joliet-long", "-R", "-V", label, "-o", "-", stage)
|
||||
var stderr bytes.Buffer
|
||||
cmd.Stderr = &stderr
|
||||
iso, err := cmd.Output()
|
||||
if err != nil {
|
||||
t.Fatalf("xorriso: %v: %s", err, stderr.String())
|
||||
}
|
||||
primary, joliet := "", ""
|
||||
for offset := 16 * 2048; offset+2048 <= len(iso); offset += 2048 {
|
||||
descriptor := iso[offset : offset+2048]
|
||||
if string(descriptor[1:6]) != "CD001" {
|
||||
t.Fatal("invalid ISO descriptor")
|
||||
}
|
||||
switch descriptor[0] {
|
||||
case 1:
|
||||
primary = strings.TrimRight(string(descriptor[40:72]), " \x00")
|
||||
case 2:
|
||||
if !bytes.HasPrefix(descriptor[88:91], []byte("%/")) {
|
||||
continue
|
||||
}
|
||||
var units [16]uint16
|
||||
for i := range units {
|
||||
units[i] = binary.BigEndian.Uint16(descriptor[40+2*i : 42+2*i])
|
||||
}
|
||||
joliet = strings.TrimRight(string(utf16.Decode(units[:])), " \x00")
|
||||
}
|
||||
if descriptor[0] == 255 {
|
||||
break
|
||||
}
|
||||
}
|
||||
if primary != label || joliet != label || label != "OTD248CEFA179C4B" {
|
||||
t.Fatalf("ISO label changed across descriptors: requested=%q primary=%q joliet=%q", label, primary, joliet)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/url"
|
||||
"os"
|
||||
"otche/internal/pve"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// OnlinePolicy refers to an operator-qualified dedicated routed sandbox. The
|
||||
// immutable proof must include management/RFC1918 denial tests and explicit
|
||||
// online/cloud-data consent; NIC firewall flags alone are never isolation proof.
|
||||
type OnlinePolicy struct {
|
||||
Bridge string `json:"bridge"`
|
||||
ProofFile string `json:"proof_file"`
|
||||
}
|
||||
type onlineProof struct {
|
||||
OwnerID string `json:"owner_id"`
|
||||
Node string `json:"node"`
|
||||
Bridge string `json:"bridge"`
|
||||
ExpiresAt time.Time `json:"expires_at"`
|
||||
Passed bool `json:"passed"`
|
||||
ManagementDenied bool `json:"management_denied"`
|
||||
PrivateNetworksDenied bool `json:"private_networks_denied"`
|
||||
CloudConsent bool `json:"cloud_consent"`
|
||||
NetworkSHA256 string `json:"network_sha256"`
|
||||
FirewallOptionsSHA256 string `json:"firewall_options_sha256"`
|
||||
FirewallRulesSHA256 string `json:"firewall_rules_sha256"`
|
||||
EvidenceSHA256 string `json:"evidence_sha256"`
|
||||
}
|
||||
|
||||
func (e *engine) checkNetwork(ctx context.Context, a attempt, b Binding, c *pve.Client, vmid int, cfg map[string]any) error {
|
||||
var opts settings
|
||||
if json.Unmarshal(a.Settings, &opts) != nil {
|
||||
return errors.New("invalid network settings")
|
||||
}
|
||||
nics := 0
|
||||
for key, value := range cfg {
|
||||
if !strings.HasPrefix(key, "net") {
|
||||
continue
|
||||
}
|
||||
nics++
|
||||
if opts.Internet == "offline" {
|
||||
return errors.New("offline disposable VM has a NIC")
|
||||
}
|
||||
if opts.Internet != "online" || b.Online == nil {
|
||||
return errors.New("online network has no configured isolation policy")
|
||||
}
|
||||
parts := map[string]string{}
|
||||
for _, part := range strings.Split(pve.Text(value), ",") {
|
||||
kv := strings.SplitN(part, "=", 2)
|
||||
if len(kv) == 2 {
|
||||
parts[kv[0]] = kv[1]
|
||||
}
|
||||
}
|
||||
if parts["bridge"] != b.Online.Bridge || parts["firewall"] != "1" || parts["link_down"] == "1" {
|
||||
return errors.New("online NIC differs from qualified sandbox")
|
||||
}
|
||||
}
|
||||
if opts.Internet == "offline" {
|
||||
return nil
|
||||
}
|
||||
if opts.Internet != "online" || nics != 1 || b.Online == nil || !ident.MatchString(b.Online.Bridge) {
|
||||
return errors.New("online requires exactly one qualified sandbox NIC")
|
||||
}
|
||||
raw, err := os.ReadFile(b.Online.ProofFile)
|
||||
if err != nil {
|
||||
return errors.New("online isolation proof missing")
|
||||
}
|
||||
var proof onlineProof
|
||||
if json.Unmarshal(raw, &proof) != nil || !proof.Passed || !proof.ManagementDenied || !proof.PrivateNetworksDenied || !proof.CloudConsent || !proof.ExpiresAt.After(time.Now()) || proof.OwnerID != a.OwnerID || proof.Node != b.Node || proof.Bridge != b.Online.Bridge || !shaRE.MatchString(proof.EvidenceSHA256) {
|
||||
return errors.New("online isolation proof invalid, expired or not consented")
|
||||
}
|
||||
var network, options map[string]any
|
||||
var rules []map[string]any
|
||||
if err = c.Do(ctx, "GET", "/nodes/"+b.Node+"/network/"+b.Online.Bridge, nil, &network); err != nil {
|
||||
return err
|
||||
}
|
||||
if pve.Text(network["active"]) != "1" && pve.Text(network["active"]) != "true" {
|
||||
return errors.New("isolated online bridge inactive")
|
||||
}
|
||||
delete(network, "digest")
|
||||
if digest(network) != proof.NetworkSHA256 {
|
||||
return errors.New("isolated network topology drift")
|
||||
}
|
||||
path := pve.VMPath(b.Node, vmid) + "/firewall"
|
||||
if err = c.Do(ctx, "GET", path+"/options", nil, &options); err != nil {
|
||||
return err
|
||||
}
|
||||
delete(options, "digest")
|
||||
if pve.Text(options["enable"]) != "1" && pve.Text(options["enable"]) != "true" {
|
||||
return errors.New("online VM firewall not enabled")
|
||||
}
|
||||
if pve.Text(options["policy_in"]) != "DROP" || pve.Text(options["policy_out"]) != "DROP" {
|
||||
return errors.New("online default firewall policies are not deny")
|
||||
}
|
||||
if err = c.Do(ctx, "GET", path+"/rules", url.Values{}, &rules); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, r := range rules {
|
||||
delete(r, "digest")
|
||||
}
|
||||
if digest(options) != proof.FirewallOptionsSHA256 || digest(rules) != proof.FirewallRulesSHA256 {
|
||||
return errors.New("online firewall differs from qualified enforcement policy")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"otche/internal/pve"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestNetworkGateRejectsUnprovenOnlineAndOfflineNIC(t *testing.T) {
|
||||
network := map[string]any{"active": true, "iface": "isolated", "type": "bridge"}
|
||||
options := map[string]any{"enable": true, "policy_in": "DROP", "policy_out": "DROP"}
|
||||
rules := []map[string]any{{"type": "out", "action": "ACCEPT", "dest": "198.51.100.1", "enable": 1}}
|
||||
server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
var result any
|
||||
switch {
|
||||
case strings.Contains(r.URL.Path, "/network/"):
|
||||
result = network
|
||||
case strings.HasSuffix(r.URL.Path, "/options"):
|
||||
result = options
|
||||
case strings.HasSuffix(r.URL.Path, "/rules"):
|
||||
result = rules
|
||||
default:
|
||||
t.Errorf("unexpected %s", r.URL.Path)
|
||||
}
|
||||
json.NewEncoder(w).Encode(map[string]any{"data": result})
|
||||
}))
|
||||
defer server.Close()
|
||||
dir := t.TempDir()
|
||||
ca := filepath.Join(dir, "ca.pem")
|
||||
credential := filepath.Join(dir, "credential.json")
|
||||
if err := os.WriteFile(ca, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw}), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(credential, []byte(`{"token_id":"fixture@pve!test","secret":"test-only"}`), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
client, err := pve.New(server.URL, ca, credential)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer client.Close()
|
||||
e := engine{}
|
||||
a := attempt{OwnerID: newID(), Settings: json.RawMessage(`{"internet":"offline"}`)}
|
||||
b := Binding{Node: "node"}
|
||||
nic := map[string]any{"net0": "virtio=12:34:56:78:90:AB,bridge=isolated,firewall=1"}
|
||||
if e.checkNetwork(context.Background(), a, b, client, 9001, nic) == nil {
|
||||
t.Fatal("offline NIC was accepted")
|
||||
}
|
||||
if err = e.checkNetwork(context.Background(), a, b, client, 9001, map[string]any{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a.Settings = json.RawMessage(`{"internet":"online"}`)
|
||||
if e.checkNetwork(context.Background(), a, b, client, 9001, nic) == nil {
|
||||
t.Fatal("unconfigured online was accepted")
|
||||
}
|
||||
proofPath := filepath.Join(dir, "proof.json")
|
||||
b.Online = &OnlinePolicy{Bridge: "isolated", ProofFile: proofPath}
|
||||
proof := onlineProof{OwnerID: a.OwnerID, Node: b.Node, Bridge: "isolated", Passed: true, ExpiresAt: time.Now().Add(time.Hour), ManagementDenied: true, PrivateNetworksDenied: true, CloudConsent: true, NetworkSHA256: digest(network), FirewallOptionsSHA256: digest(options), FirewallRulesSHA256: digest(rules), EvidenceSHA256: strings.Repeat("a", 64)}
|
||||
if err = atomicJSON(proofPath, proof); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = e.checkNetwork(context.Background(), a, b, client, 9001, nic); err != nil {
|
||||
t.Fatalf("matching qualified enforcement rejected: %v", err)
|
||||
}
|
||||
options["policy_out"] = "ACCEPT"
|
||||
if e.checkNetwork(context.Background(), a, b, client, 9001, nic) == nil {
|
||||
t.Fatal("firewall drift to default allow accepted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// salvagePartial preserves whatever was durably flushed without turning a
|
||||
// missing final result into clean telemetry. It never schedules guest code.
|
||||
func (e *engine) salvagePartial(ctx context.Context, a attempt, l allocation, b Binding, cs clients, dir string) (json.RawMessage, string) {
|
||||
var report json.RawMessage
|
||||
findings := "unknown"
|
||||
if e.leaseAlive(ctx, a) != nil || cs.provisioner.CheckOwned(ctx, e.own(a, l, b)) != nil {
|
||||
return nil, findings
|
||||
}
|
||||
salvageDir := filepath.Join(dir, "salvage-"+newID())
|
||||
if os.MkdirAll(salvageDir, 0700) != nil {
|
||||
return nil, findings
|
||||
}
|
||||
var total int64
|
||||
for _, name := range []string{"result.json", "status.json", "receipt.json", "telemetry.json", "defender-events.json"} {
|
||||
if total >= e.cfg.MaxArtifactBytes {
|
||||
return report, findings
|
||||
}
|
||||
var raw bytes.Buffer
|
||||
remaining := e.cfg.MaxArtifactBytes - total
|
||||
if remaining > 4<<20 {
|
||||
remaining = 4 << 20
|
||||
}
|
||||
n, err := cs.runtime.ReadFile(ctx, b.Node, l.VMID, guestDir(a)+name, &raw, remaining)
|
||||
total += n
|
||||
if err != nil || !json.Valid(raw.Bytes()) {
|
||||
continue
|
||||
}
|
||||
path := filepath.Join(salvageDir, name)
|
||||
if err = os.WriteFile(path, raw.Bytes(), 0600); err != nil {
|
||||
continue
|
||||
}
|
||||
_ = e.publishFile(ctx, a, "partial_telemetry", name, "application/json", path)
|
||||
if name == "result.json" || name == "status.json" {
|
||||
var r guestResult
|
||||
if json.Unmarshal(raw.Bytes(), &r) == nil && r.CommandID == a.CommandID && r.AttemptID == a.ID && r.JobID == a.JobID {
|
||||
if json.Valid(r.Report) {
|
||||
report = r.Report
|
||||
}
|
||||
if r.Findings == "detected" {
|
||||
findings = "detected"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return report, findings
|
||||
}
|
||||
func appendObservation(path string, v json.RawMessage, limit int64) error {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return err
|
||||
}
|
||||
size := int64(0)
|
||||
if info != nil {
|
||||
size = info.Size()
|
||||
}
|
||||
if size+int64(len(v))+1 > limit {
|
||||
return errors.New("bounded external observation journal full")
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = f.Write(append(append([]byte(nil), v...), '\n'))
|
||||
if err == nil {
|
||||
err = f.Sync()
|
||||
}
|
||||
closeErr := f.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return closeErr
|
||||
}
|
||||
func salvageExtracted(dir string, a attempt) (json.RawMessage, string) {
|
||||
raw, err := os.ReadFile(filepath.Join(dir, "extraction-manifest.json"))
|
||||
if err != nil || len(raw) > 1<<20 {
|
||||
return nil, "unknown"
|
||||
}
|
||||
var m struct {
|
||||
CommandID string `json:"command_id"`
|
||||
Files []struct {
|
||||
Name string `json:"name"`
|
||||
Size int64 `json:"size"`
|
||||
} `json:"files"`
|
||||
}
|
||||
if json.Unmarshal(raw, &m) != nil || m.CommandID != a.CommandID || len(m.Files) > 64 {
|
||||
return nil, "unknown"
|
||||
}
|
||||
for _, f := range m.Files {
|
||||
if filepath.Base(f.Name) != f.Name || len(f.Name) < 12 || f.Name[len(f.Name)-12:] != "-result.json" || f.Size < 1 || f.Size > 4<<20 {
|
||||
continue
|
||||
}
|
||||
path := filepath.Join(dir, "extracted-"+f.Name)
|
||||
info, err := os.Lstat(path)
|
||||
if err != nil || !info.Mode().IsRegular() || info.Size() != f.Size {
|
||||
continue
|
||||
}
|
||||
result, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var r guestResult
|
||||
if json.Unmarshal(result, &r) == nil && validResult(a, r) {
|
||||
return r.Report, r.Findings
|
||||
}
|
||||
}
|
||||
return nil, "unknown"
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestExtractedReportRejectsDifferentCommand(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
a := attempt{ID: newID(), JobID: newID(), CommandID: newID()}
|
||||
r := guestResult{AttemptID: a.ID, JobID: a.JobID, CommandID: newID(), Outcome: "executed", Findings: "detected", Telemetry: "complete", Report: json.RawMessage(`{"execution":{"exit_code":0}}`)}
|
||||
raw, _ := json.Marshal(r)
|
||||
name := "000-result.json"
|
||||
path := filepath.Join(dir, "extracted-"+name)
|
||||
if err := os.WriteFile(path, raw, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
manifest := map[string]any{"command_id": a.CommandID, "files": []map[string]any{{"name": name, "size": len(raw)}}}
|
||||
if err := atomicJSON(filepath.Join(dir, "extraction-manifest.json"), manifest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if report, _ := salvageExtracted(dir, a); report != nil {
|
||||
t.Fatal("foreign command result crossed attempt boundary")
|
||||
}
|
||||
r.CommandID = a.CommandID
|
||||
raw, _ = json.Marshal(r)
|
||||
if err := os.WriteFile(path, raw, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if report, findings := salvageExtracted(dir, a); report == nil || findings != "detected" {
|
||||
t.Fatal("same command observed detection was lost")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,233 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"net/url"
|
||||
"otche/internal/pve"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func isNoRows(err error) bool { return errors.Is(err, pgx.ErrNoRows) }
|
||||
func (e *engine) provision(ctx context.Context, a attempt, b Binding, cs clients) (allocation, error) {
|
||||
s, ok := e.cfg.Sources[a.SourceRef]
|
||||
if !ok || s.Node != b.Node {
|
||||
return allocation{}, errors.New("configured same-node stopped source is required")
|
||||
}
|
||||
conn, err := sourceLock(ctx, e.db, a.SourceRef)
|
||||
if err != nil {
|
||||
return allocation{}, err
|
||||
}
|
||||
defer unlockSource(conn, a.SourceRef)
|
||||
if err = checkSeal(s, a.SourceRef, a.RevisionID, a.ConfigDigest); err != nil {
|
||||
return allocation{}, err
|
||||
}
|
||||
source, dg, err := sourceConfig(ctx, cs.provisioner, s)
|
||||
if err != nil {
|
||||
return allocation{}, err
|
||||
}
|
||||
if dg != a.ConfigDigest {
|
||||
return allocation{}, errors.New("source configuration drift")
|
||||
}
|
||||
if err = storageHeadroom(ctx, cs.provisioner, b, s.MaxDiskBytes); err != nil {
|
||||
return allocation{}, err
|
||||
}
|
||||
var used int64
|
||||
if err = e.db.QueryRow(ctx, `SELECT COALESCE(sum(bytes),0) FROM (SELECT COALESCE((metadata->>'reserved_disk_bytes')::bigint,0) bytes FROM allocations WHERE owner_id=$1 AND state<>'deleted' UNION ALL SELECT COALESCE((metadata->>'reserved_disk_bytes')::bigint,0) FROM extractor_allocations WHERE owner_id=$1 AND state<>'deleted') reserved`, a.OwnerID).Scan(&used); err != nil {
|
||||
return allocation{}, err
|
||||
}
|
||||
reserve := s.MaxDiskBytes
|
||||
if e.cfg.Extractor != nil {
|
||||
reserve += e.cfg.Extractor.MaxDiskBytes
|
||||
}
|
||||
if used+reserve > b.MaxOwnedDiskBytes {
|
||||
return allocation{}, errors.New("owner disk reservation quota exceeded including retained evidence/extractor")
|
||||
}
|
||||
l, err := e.getAllocation(ctx, a)
|
||||
if err == nil {
|
||||
return l, errors.New("allocation already exists: reconcile, never issue another clone")
|
||||
}
|
||||
if !isNoRows(err) {
|
||||
return l, err
|
||||
}
|
||||
idConn, err := sourceLock(ctx, e.db, "nextid-"+b.Endpoint)
|
||||
if err != nil {
|
||||
return l, err
|
||||
}
|
||||
defer unlockSource(idConn, "nextid-"+b.Endpoint)
|
||||
var suggested string
|
||||
if err = cs.provisioner.Do(ctx, "GET", "/cluster/nextid", nil, &suggested); err != nil {
|
||||
return l, err
|
||||
}
|
||||
id, err := strconv.Atoi(suggested)
|
||||
if err != nil {
|
||||
return l, errors.New("invalid PVE nextid")
|
||||
}
|
||||
for tries := range 64 {
|
||||
blocked := false
|
||||
for _, n := range e.cfg.protected() {
|
||||
if id == n {
|
||||
blocked = true
|
||||
}
|
||||
}
|
||||
var exists bool
|
||||
if err = e.db.QueryRow(ctx, "SELECT EXISTS(SELECT 1 FROM allocations WHERE vmid=$1 AND state<>'deleted') OR EXISTS(SELECT 1 FROM extractor_allocations WHERE vmid=$1 AND state<>'deleted')", id).Scan(&exists); err != nil {
|
||||
return l, err
|
||||
}
|
||||
if !blocked && !exists {
|
||||
break
|
||||
}
|
||||
if err = cs.provisioner.Do(ctx, "GET", "/cluster/nextid", url.Values{"vmid": {strconv.Itoa(id + 1)}}, &suggested); err != nil {
|
||||
return l, err
|
||||
}
|
||||
id, err = strconv.Atoi(suggested)
|
||||
if err != nil {
|
||||
return l, err
|
||||
}
|
||||
if tries == 63 {
|
||||
return l, errors.New("could not reserve next free VMID")
|
||||
}
|
||||
}
|
||||
l = allocation{ID: newID(), VMID: id, State: "reserved"}
|
||||
metadata, _ := json.Marshal(map[string]any{"reserved_disk_bytes": s.MaxDiskBytes, "artifact_reservation": e.cfg.MaxVideoBytes + 3*e.cfg.MaxArtifactBytes + grubReservation(a, e.cfg.MaxArtifactBytes), "source_ref": a.SourceRef, "source_config": source, "source_vmid": s.VMID, "cd_slot": s.CDSlot})
|
||||
_, err = e.db.Exec(ctx, "INSERT INTO allocations(id,attempt_id,owner_id,node,vmid,pool,state,metadata) VALUES($1,$2,$3,$4,$5,$6,'reserved',$7)", l.ID, a.ID, a.OwnerID, b.Node, id, b.Pool, metadata)
|
||||
if err != nil {
|
||||
return l, err
|
||||
}
|
||||
l.Metadata = metadata
|
||||
if err = e.recordAction(ctx, a, l, "clone_intent", ""); err != nil {
|
||||
return l, err
|
||||
}
|
||||
taskCtx, cancel := context.WithTimeout(ctx, time.Duration(e.cfg.TaskTimeoutSeconds)*time.Second)
|
||||
defer cancel()
|
||||
var upid string
|
||||
err = cs.provisioner.Do(taskCtx, "POST", pve.VMPath(s.Node, s.VMID)+"/clone", url.Values{"newid": {strconv.Itoa(id)}, "full": {"1"}, "pool": {b.Pool}, "name": {"otche-" + a.ID}, "storage": {b.DiskStorage}}, &upid)
|
||||
if err != nil {
|
||||
return l, err
|
||||
}
|
||||
if err = pve.ValidateUPID(s.Node, upid); err != nil {
|
||||
return l, err
|
||||
}
|
||||
l.UPID = &upid
|
||||
if err = e.recordAction(ctx, a, l, "cloning", upid); err != nil {
|
||||
return l, err
|
||||
}
|
||||
if err = cs.provisioner.Task(taskCtx, s.Node, upid); err != nil {
|
||||
return l, err
|
||||
}
|
||||
_, after, err := sourceConfig(taskCtx, cs.provisioner, s)
|
||||
if err != nil || after != dg {
|
||||
return l, errors.New("source changed during full clone; disposable evidence held")
|
||||
}
|
||||
if err = checkSeal(s, a.SourceRef, a.RevisionID, a.ConfigDigest); err != nil {
|
||||
return l, err
|
||||
}
|
||||
if err = e.adoptCompletedClone(taskCtx, a, l, b, cs); err != nil {
|
||||
return l, err
|
||||
}
|
||||
l.State = "cloned"
|
||||
return l, nil
|
||||
}
|
||||
func (e *engine) prepareClone(ctx context.Context, a attempt, l allocation, b Binding, cs clients) error {
|
||||
o := e.own(a, l, b)
|
||||
state, err := cs.provisioner.Status(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if state != "stopped" {
|
||||
return errors.New("clone must be stopped before hardware preparation")
|
||||
}
|
||||
cfg, err := cs.provisioner.Config(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var opts settings
|
||||
if json.Unmarshal(a.Settings, &opts) != nil {
|
||||
return errors.New("invalid job settings")
|
||||
}
|
||||
if opts.Internet == "online" {
|
||||
if err = e.checkNetwork(ctx, a, b, cs.provisioner, l.VMID, cfg); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
slot := e.cfg.Sources[a.SourceRef].CDSlot
|
||||
if v, ok := cfg[slot]; ok && !cdromMatches(pve.Text(v), "none") {
|
||||
return errors.New("configured empty CD slot contains media or a non-CD device")
|
||||
}
|
||||
v := url.Values{slot: {"none,media=cdrom"}, "onboot": {"0"}}
|
||||
var remove []string
|
||||
if opts.Internet == "offline" {
|
||||
for k := range cfg {
|
||||
if strings.HasPrefix(k, "net") {
|
||||
remove = append(remove, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(remove) > 0 {
|
||||
v.Set("delete", strings.Join(remove, ","))
|
||||
}
|
||||
if err = e.fence(ctx, a); err != nil {
|
||||
return err
|
||||
}
|
||||
upid, err := cs.provisioner.OwnedTask(ctx, o, "POST", "/config", v)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.recordAction(ctx, a, l, "configuring", upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
cfg, err = cs.provisioner.Config(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.checkNetwork(ctx, a, b, cs.provisioner, l.VMID, cfg); err != nil {
|
||||
return err
|
||||
}
|
||||
if !cdromMatches(pve.Text(cfg[slot]), "none") {
|
||||
return errors.New("empty CD verification failed")
|
||||
}
|
||||
return cs.provisioner.NoPending(ctx, b.Node, l.VMID, "")
|
||||
}
|
||||
func (e *engine) swapISO(ctx context.Context, a attempt, l allocation, b Binding, cs clients, volume string) error {
|
||||
if err := e.fence(ctx, a); err != nil {
|
||||
return err
|
||||
}
|
||||
o := e.own(a, l, b)
|
||||
slot := e.cfg.Sources[a.SourceRef].CDSlot
|
||||
cfg, err := cs.provisioner.Config(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !cdromMatches(pve.Text(cfg[slot]), "none") {
|
||||
return errors.New("hot media insertion requires the existing empty drive")
|
||||
}
|
||||
if err = e.checkNetwork(ctx, a, b, cs.provisioner, l.VMID, cfg); err != nil {
|
||||
return err
|
||||
}
|
||||
upid, err := cs.provisioner.OwnedTask(ctx, o, "POST", "/config", url.Values{slot: {volume + ",media=cdrom"}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = e.recordAction(ctx, a, l, "inserting", upid); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
cfg, err = cs.provisioner.Config(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !cdromMatches(pve.Text(cfg[slot]), volume) {
|
||||
return fmt.Errorf("hot CD current configuration did not match job ISO")
|
||||
}
|
||||
return cs.provisioner.NoPending(ctx, b.Node, l.VMID, slot)
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type qualificationControl struct {
|
||||
Kind string `json:"kind"`
|
||||
Internet string `json:"internet"`
|
||||
JobID string `json:"job_id"`
|
||||
AttemptID string `json:"attempt_id"`
|
||||
}
|
||||
type qualificationResult struct {
|
||||
RevisionID string `json:"revision_id"`
|
||||
Controls []qualificationControl `json:"controls"`
|
||||
Errors []string `json:"errors"`
|
||||
Baseline string `json:"baseline_fingerprint,omitempty"`
|
||||
}
|
||||
|
||||
func (e *engine) qualify(ctx context.Context) error {
|
||||
// Resume already queued controls; never create replacements after a restart.
|
||||
rows, err := e.db.Query(ctx, "SELECT id,profile_id,result FROM qualifications WHERE status='running' ORDER BY created_at LIMIT 8")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
type active struct {
|
||||
id, profile string
|
||||
raw []byte
|
||||
}
|
||||
var running []active
|
||||
for rows.Next() {
|
||||
var v active
|
||||
if err = rows.Scan(&v.id, &v.profile, &v.raw); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
running = append(running, v)
|
||||
}
|
||||
rows.Close()
|
||||
if err = rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, v := range running {
|
||||
if err = e.completeQualification(ctx, v.id, v.profile, v.raw); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
tx, err := e.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
var id, profile, revision, ref, dg string
|
||||
err = tx.QueryRow(ctx, `SELECT q.id,q.profile_id,COALESCE(p.current_revision_id::text,''),COALESCE(v.source_ref,''),COALESCE(v.config_digest,'') FROM qualifications q JOIN profiles p ON p.id=q.profile_id LEFT JOIN revisions v ON v.id=p.current_revision_id WHERE q.status='queued' ORDER BY q.created_at FOR UPDATE OF q SKIP LOCKED LIMIT 1`).Scan(&id, &profile, &revision, &ref, &dg)
|
||||
if isNoRows(err) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
result := qualificationResult{RevisionID: revision, Controls: []qualificationControl{}, Errors: []string{}}
|
||||
owner := e.cfg.QualificationOwner
|
||||
b, ok := e.cfg.Owners[owner]
|
||||
source, sourceOK := e.cfg.Sources[ref]
|
||||
if !ok || !sourceOK || revision == "" {
|
||||
result.Errors = append(result.Errors, "Worker qualification owner and validated current stopped source are required")
|
||||
} else if err = b.proof(owner); err != nil {
|
||||
result.Errors = append(result.Errors, err.Error())
|
||||
} else if err = checkSeal(source, ref, revision, dg); err != nil {
|
||||
result.Errors = append(result.Errors, err.Error())
|
||||
}
|
||||
if !shaRE.MatchString(e.cfg.BenignSHA256) || !filepath.IsAbs(e.cfg.BenignFixture) || !strings.EqualFold(filepath.Ext(e.cfg.BenignFixture), ".exe") {
|
||||
result.Errors = append(result.Errors, "Explicit benign native EXE fixture file and verified SHA-256 required")
|
||||
}
|
||||
if len(result.Errors) == 0 {
|
||||
networks := []string{"offline"}
|
||||
if b.Online != nil {
|
||||
networks = append(networks, "online")
|
||||
}
|
||||
for _, internet := range networks {
|
||||
for _, kind := range []string{"eicar", "benign"} {
|
||||
control, err := e.enqueueControl(ctx, tx, id, profile, revision, owner, kind, internet)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
result.Controls = append(result.Controls, control)
|
||||
}
|
||||
}
|
||||
}
|
||||
status := "running"
|
||||
if len(result.Errors) > 0 {
|
||||
status = "failed"
|
||||
}
|
||||
raw, _ := json.Marshal(result)
|
||||
_, err = tx.Exec(ctx, "UPDATE qualifications SET status=$2,result=$3 WHERE id=$1", id, status, raw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
func (e *engine) enqueueControl(ctx context.Context, tx pgx.Tx, qid, profile, revision, owner, kind, internet string) (qualificationControl, error) {
|
||||
c := qualificationControl{Kind: kind, Internet: internet, JobID: newID(), AttemptID: newID()}
|
||||
upload, run, command := newID(), newID(), newID()
|
||||
filename := "otche-benign.exe"
|
||||
if kind == "eicar" {
|
||||
filename = "eicar.com"
|
||||
}
|
||||
key := filepath.ToSlash(filepath.Join("qualification", qid, internet, kind, filename))
|
||||
path, err := contained(e.root, key)
|
||||
if err != nil {
|
||||
return c, err
|
||||
}
|
||||
if err = os.MkdirAll(filepath.Dir(path), 0700); err != nil {
|
||||
return c, err
|
||||
}
|
||||
out, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return c, errors.New("qualification fixture exists or is unavailable; no overwrite")
|
||||
}
|
||||
h := sha256.New()
|
||||
dst := io.MultiWriter(out, h)
|
||||
var n int64
|
||||
if kind == "eicar" {
|
||||
n, err = io.Copy(dst, strings.NewReader("X5O!P%@AP[4\\PZX54(P^)7CC)7}$"+"EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*"))
|
||||
} else {
|
||||
var src *os.File
|
||||
src, err = os.Open(e.cfg.BenignFixture)
|
||||
if err == nil {
|
||||
n, err = io.Copy(dst, io.LimitReader(src, 64<<20+1))
|
||||
src.Close()
|
||||
if n > 64<<20 {
|
||||
err = errors.New("benign fixture exceeds 64 MiB bound")
|
||||
}
|
||||
}
|
||||
}
|
||||
syncErr := out.Sync()
|
||||
closeErr := out.Close()
|
||||
if err != nil || syncErr != nil || closeErr != nil {
|
||||
return c, errors.New("qualification fixture creation failed")
|
||||
}
|
||||
hash := hex.EncodeToString(h.Sum(nil))
|
||||
if kind == "benign" && hash != e.cfg.BenignSHA256 {
|
||||
return c, errors.New("benign fixture SHA-256 differs from configured known-safe control")
|
||||
}
|
||||
settings, _ := json.Marshal(map[string]any{"internet": internet, "duration_seconds": 30, "filename": "original", "privilege": "user", "args_mode": "none", "args": []string{}, "set_zoneid": false, "dll_mode": "regsvr32", "dll_export": "", "architecture": "auto", "wsh_host": "cscript", "msi_ui": "full", "_qualification": kind})
|
||||
if _, err = tx.Exec(ctx, "INSERT INTO uploads(id,owner_id,filename,size,sha256,storage_key) VALUES($1,$2,$3,$4,$5,$6)", upload, owner, filename, n, hash, key); err != nil {
|
||||
return c, err
|
||||
}
|
||||
if _, err = tx.Exec(ctx, "INSERT INTO jobs(id,owner_id,upload_id,execution_filename,settings) VALUES($1,$2,$3,$4,$5)", c.JobID, owner, upload, filename, settings); err != nil {
|
||||
return c, err
|
||||
}
|
||||
if _, err = tx.Exec(ctx, "INSERT INTO runs(id,job_id,profile_id,revision_id) VALUES($1,$2,$3,$4)", run, c.JobID, profile, revision); err != nil {
|
||||
return c, err
|
||||
}
|
||||
_, err = tx.Exec(ctx, "INSERT INTO attempts(id,run_id,command_id) VALUES($1,$2,$3)", c.AttemptID, run, command)
|
||||
return c, err
|
||||
}
|
||||
func (e *engine) completeQualification(ctx context.Context, id, profile string, raw []byte) error {
|
||||
var result qualificationResult
|
||||
if json.Unmarshal(raw, &result) != nil || (len(result.Controls) != 2 && len(result.Controls) != 4) {
|
||||
return errors.New("qualification durable control manifest is invalid")
|
||||
}
|
||||
baseline := ""
|
||||
var metadata any
|
||||
var environment json.RawMessage
|
||||
onlineControls := map[string]bool{}
|
||||
for _, control := range result.Controls {
|
||||
if control.Internet == "online" {
|
||||
onlineControls[control.Kind] = true
|
||||
}
|
||||
}
|
||||
for _, control := range result.Controls {
|
||||
var phase, outcome, findings, telemetry, cleanup string
|
||||
var report []byte
|
||||
if err := e.db.QueryRow(ctx, "SELECT phase,outcome,findings,telemetry,cleanup,report FROM attempts WHERE id=$1", control.AttemptID).Scan(&phase, &outcome, &findings, &telemetry, &cleanup, &report); err != nil {
|
||||
return err
|
||||
}
|
||||
if phase != "finished" {
|
||||
return nil
|
||||
}
|
||||
if telemetry != "complete" || cleanup != "complete" {
|
||||
result.Errors = append(result.Errors, control.Kind+" control observation/collection/cleanup incomplete")
|
||||
}
|
||||
if control.Kind == "eicar" && (findings != "detected" || outcome != "blocked_before_execution") {
|
||||
result.Errors = append(result.Errors, "EICAR delivery-only detection control did not pass")
|
||||
}
|
||||
if control.Kind == "benign" && (outcome != "executed" || findings != "not_observed") {
|
||||
result.Errors = append(result.Errors, "Benign executable control did not pass")
|
||||
}
|
||||
var r struct {
|
||||
Execution struct {
|
||||
Duration *float64 `json:"actual_duration_seconds"`
|
||||
ExitCode *int `json:"exit_code"`
|
||||
} `json:"execution"`
|
||||
Defender struct {
|
||||
Before, After *struct {
|
||||
Active bool `json:"active"`
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
}
|
||||
Drift bool `json:"drift"`
|
||||
} `json:"defender"`
|
||||
CollectionErrors []string `json:"collection_errors"`
|
||||
}
|
||||
if json.Unmarshal(report, &r) != nil || r.Defender.Before == nil || r.Defender.After == nil || !r.Defender.Before.Active || !r.Defender.After.Active || r.Defender.Drift || r.Defender.Before.Fingerprint != r.Defender.After.Fingerprint || len(r.CollectionErrors) > 0 {
|
||||
result.Errors = append(result.Errors, control.Kind+" baseline/evidence changed or is unavailable")
|
||||
}
|
||||
if control.Kind == "benign" && (r.Execution.Duration == nil || *r.Execution.Duration < 30 || r.Execution.ExitCode == nil || *r.Execution.ExitCode != 0) {
|
||||
result.Errors = append(result.Errors, "Benign control did not complete actual timer with successful exit")
|
||||
}
|
||||
var key string
|
||||
if err := e.db.QueryRow(ctx, "SELECT storage_key FROM artifacts WHERE attempt_id=$1 AND kind='baseline'", control.AttemptID).Scan(&key); err != nil {
|
||||
result.Errors = append(result.Errors, "Observed baseline artifact missing")
|
||||
continue
|
||||
}
|
||||
path, err := contained(e.root, key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
observed, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var ready guestReady
|
||||
if json.Unmarshal(observed, &ready) != nil || !ready.Baseline.Qualified || !shaRE.MatchString(ready.Baseline.Fingerprint) {
|
||||
result.Errors = append(result.Errors, "Observed baseline is not readable/active")
|
||||
continue
|
||||
}
|
||||
if baseline != "" && baseline != ready.Baseline.Fingerprint {
|
||||
result.Errors = append(result.Errors, "Control clones did not share identical source/Defender baseline")
|
||||
}
|
||||
baseline = ready.Baseline.Fingerprint
|
||||
environment = ready.Environment
|
||||
metadata = map[string]any{"environment": json.RawMessage(ready.Environment), "defender": json.RawMessage(ready.Defender), "baseline_fingerprint": baseline}
|
||||
}
|
||||
status, state := "passed", "qualified"
|
||||
if len(result.Errors) > 0 {
|
||||
status, state = "failed", "unqualified"
|
||||
}
|
||||
result.Baseline = baseline
|
||||
encoded, _ := json.Marshal(result)
|
||||
q, _ := json.Marshal(map[string]any{"worker_validated": true, "status": status, "state": state, "baseline_fingerprint": baseline, "qualification_id": id, "controls": result.Controls, "errors": result.Errors, "environment": environment})
|
||||
meta, _ := json.Marshal(metadata)
|
||||
tx, err := e.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
if _, err = tx.Exec(ctx, "UPDATE qualifications SET status=$2,result=$3 WHERE id=$1 AND status='running'", id, status, encoded); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = tx.Exec(ctx, "UPDATE revisions SET qualification=$2 WHERE id=$1", result.RevisionID, q); err != nil {
|
||||
return err
|
||||
}
|
||||
reason := "Observed qualification passed; explicit admin publication required"
|
||||
if status == "failed" {
|
||||
reason = "Qualification controls failed; inspect preserved evidence"
|
||||
}
|
||||
if _, err = tx.Exec(ctx, "UPDATE profiles SET qualification=$2,enabled=false,state='maintenance',reason=$3,metadata=$4,online_available=$6 WHERE id=$1 AND current_revision_id=$5", profile, state, reason, meta, result.RevisionID, status == "passed" && onlineControls["eicar"] && onlineControls["benign"]); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
@@ -0,0 +1,316 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"net/url"
|
||||
"otche/internal/pve"
|
||||
"time"
|
||||
)
|
||||
|
||||
func vmAbsent(ctx context.Context, c *pve.Client, node string, id int) (bool, error) {
|
||||
var vms []struct {
|
||||
VMID int `json:"vmid"`
|
||||
}
|
||||
if err := c.Do(ctx, "GET", "/nodes/"+node+"/qemu", nil, &vms); err != nil {
|
||||
return false, err
|
||||
}
|
||||
for _, v := range vms {
|
||||
if v.VMID == id {
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
func (e *engine) leaseAlive(ctx context.Context, a attempt) error {
|
||||
var ok bool
|
||||
if err := e.db.QueryRow(ctx, "SELECT COALESCE(lease_owner=$2 AND lease_until>now(),false) FROM attempts WHERE id=$1", a.ID, e.id).Scan(&ok); err != nil {
|
||||
return err
|
||||
}
|
||||
if !ok {
|
||||
return errors.New("attempt lease generation expired or superseded")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (e *engine) stopOwned(ctx context.Context, a attempt, l allocation, b Binding, cs clients) error {
|
||||
if err := e.leaseAlive(ctx, a); err != nil {
|
||||
return err
|
||||
}
|
||||
current, err := e.getAllocation(ctx, a)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
l = current
|
||||
if l.State == "cloning" {
|
||||
if err = e.adoptCompletedClone(ctx, a, l, b, cs); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if l.State == "deleted" {
|
||||
return nil
|
||||
}
|
||||
o := e.own(a, l, b)
|
||||
if err := cs.provisioner.CheckOwned(ctx, o); err != nil {
|
||||
return errors.New("cannot establish pool/tags/name ownership for stop; resources retained")
|
||||
}
|
||||
state, err := cs.provisioner.Status(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if state == "stopped" {
|
||||
_, err = e.db.Exec(ctx, "UPDATE allocations SET state='stopped' WHERE id=$1", l.ID)
|
||||
return err
|
||||
}
|
||||
if state != "running" && state != "paused" {
|
||||
return errors.New("unexpected VM state; cannot safely stop")
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE allocations SET state='stop_intent' WHERE id=$1", l.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
upid, err := cs.provisioner.OwnedTask(ctx, o, "POST", "/status/stop", nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE allocations SET state='stopping',upid=$2 WHERE id=$1", l.ID, upid)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
state, err = cs.provisioner.Status(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if state != "stopped" {
|
||||
return errors.New("PVE stop task did not stop clone")
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE allocations SET state='stopped' WHERE id=$1", l.ID)
|
||||
return err
|
||||
}
|
||||
func (e *engine) deleteOwned(ctx context.Context, a attempt, l allocation, b Binding, cs clients) error {
|
||||
if l.State == "deleted" {
|
||||
return nil
|
||||
}
|
||||
if err := e.leaseAlive(ctx, a); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := e.reconcileExtractor(ctx, a, l, b, cs); err != nil {
|
||||
return err
|
||||
}
|
||||
o := e.own(a, l, b)
|
||||
if err := cs.provisioner.CheckOwned(ctx, o); err != nil {
|
||||
return err
|
||||
}
|
||||
state, err := cs.provisioner.Status(ctx, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if state != "stopped" {
|
||||
return errors.New("clone destruction requires confirmed stopped state")
|
||||
}
|
||||
if err = cs.provisioner.NoPending(ctx, b.Node, l.VMID, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE allocations SET state='delete_intent' WHERE id=$1", l.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
upid, err := cs.provisioner.OwnedTask(ctx, o, "DELETE", "", url.Values{"purge": {"0"}, "destroy-unreferenced-disks": {"0"}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE allocations SET state='deleting',upid=$2 WHERE id=$1", l.ID, upid)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = cs.provisioner.Task(ctx, b.Node, upid); err != nil {
|
||||
return err
|
||||
}
|
||||
absent, err := vmAbsent(ctx, cs.provisioner, b.Node, l.VMID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !absent {
|
||||
return errors.New("clone still exists after destroy task")
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "UPDATE allocations SET state='deleted' WHERE id=$1", l.ID)
|
||||
return err
|
||||
}
|
||||
func (e *engine) releaseEvidence(ctx context.Context) error {
|
||||
rows, err := e.db.Query(ctx, "SELECT id FROM attempts WHERE phase='finished' AND release_requested AND cleanup IN ('evidence_held','failed') ORDER BY created_at LIMIT 16")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var ids []string
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err = rows.Scan(&id); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
rows.Close()
|
||||
if err = rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, id := range ids {
|
||||
a, err := e.load(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
b, ok := e.cfg.Owners[a.OwnerID]
|
||||
if !ok {
|
||||
return errors.New("retained owner binding is unavailable")
|
||||
}
|
||||
cs, err := b.clients()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
action, cancel := context.WithTimeout(ctx, 120*time.Second)
|
||||
tag, err := e.db.Exec(action, "UPDATE attempts SET lease_owner=$2,lease_until=now()+interval '150 seconds' WHERE id=$1 AND (lease_until IS NULL OR lease_until<now())", a.ID, e.id)
|
||||
if err != nil {
|
||||
cancel()
|
||||
cs.close()
|
||||
return err
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
cancel()
|
||||
cs.close()
|
||||
continue
|
||||
}
|
||||
l, err := e.getAllocation(action, a)
|
||||
if err == nil && (l.State == "deleting" || l.State == "delete_intent") {
|
||||
if l.UPID != nil && l.State == "deleting" {
|
||||
err = cs.provisioner.Task(action, b.Node, *l.UPID)
|
||||
}
|
||||
if err == nil {
|
||||
var absent bool
|
||||
absent, err = vmAbsent(action, cs.provisioner, b.Node, l.VMID)
|
||||
if err == nil && absent {
|
||||
_, err = e.db.Exec(action, "UPDATE allocations SET state='deleted' WHERE id=$1", l.ID)
|
||||
l.State = "deleted"
|
||||
}
|
||||
}
|
||||
}
|
||||
if err == nil && l.State != "deleted" {
|
||||
err = e.stopOwned(action, a, l, b, cs)
|
||||
if err == nil {
|
||||
err = e.deleteOwned(action, a, l, b, cs)
|
||||
}
|
||||
}
|
||||
if err == nil {
|
||||
err = e.removeMedia(action, a, b, cs)
|
||||
}
|
||||
if err == nil {
|
||||
_, err = e.db.Exec(action, "UPDATE attempts SET cleanup='complete',release_requested=false,lease_until=NULL WHERE id=$1 AND lease_owner=$2 AND lease_until>now()", a.ID, e.id)
|
||||
} else {
|
||||
_, _ = e.db.Exec(action, "UPDATE attempts SET cleanup='failed',error=$2,lease_until=NULL WHERE id=$1 AND lease_owner=$3 AND lease_until>now()", a.ID, err.Error(), e.id)
|
||||
}
|
||||
cancel()
|
||||
cs.close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Watchdog must be a separate supervised process. It does not record, dispatch,
|
||||
// clone or delete. It takes over expired leases/deadlines and only stops verified
|
||||
// owned disposable VMs, retaining evidence for recovery or explicit release.
|
||||
func Watchdog(ctx context.Context, db *pgxpool.Pool, configPath string) error {
|
||||
cfg, err := loadConfig(configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
e := &engine{db: db, cfg: cfg, id: "watchdog-" + newID()}
|
||||
tick := time.NewTicker(5 * time.Second)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
blockers := []string{}
|
||||
rows, err := db.Query(ctx, `SELECT a.id FROM attempts a JOIN allocations l ON l.attempt_id=a.id WHERE (l.state NOT IN ('deleted','evidence_held','stopped') OR EXISTS(SELECT 1 FROM extractor_allocations x WHERE x.attempt_id=a.id AND x.state<>'deleted')) AND (a.lease_until IS NULL OR a.lease_until<now() OR (l.metadata->>'safety_deadline')::timestamptz<now()) ORDER BY a.created_at LIMIT 32`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var ids []string
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err = rows.Scan(&id); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
rows.Close()
|
||||
if err = rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, id := range ids {
|
||||
action, cancel := context.WithTimeout(ctx, 90*time.Second)
|
||||
a, err := e.load(action, id)
|
||||
if err != nil {
|
||||
cancel()
|
||||
blockers = append(blockers, "Watchdog attempt lookup failed")
|
||||
continue
|
||||
}
|
||||
b, ok := cfg.Owners[a.OwnerID]
|
||||
if !ok {
|
||||
cancel()
|
||||
blockers = append(blockers, "Watchdog owner binding unavailable")
|
||||
continue
|
||||
}
|
||||
cs, err := b.clients()
|
||||
if err != nil {
|
||||
cancel()
|
||||
blockers = append(blockers, err.Error())
|
||||
continue
|
||||
}
|
||||
generation := *e
|
||||
generation.id = newID()
|
||||
tag, claimErr := db.Exec(action, "UPDATE attempts SET lease_owner=$2,lease_until=now()+interval '120 seconds' WHERE id=$1 AND (lease_until IS NULL OR lease_until<now() OR EXISTS(SELECT 1 FROM allocations WHERE attempt_id=$1 AND (metadata->>'safety_deadline')::timestamptz<now()))", id, generation.id)
|
||||
err = claimErr
|
||||
if err == nil && tag.RowsAffected() == 0 {
|
||||
cs.close()
|
||||
cancel()
|
||||
continue
|
||||
}
|
||||
if err == nil {
|
||||
var l allocation
|
||||
l, err = e.getAllocation(action, a)
|
||||
if err == nil {
|
||||
err = generation.stopOwned(action, a, l, b, cs)
|
||||
if err == nil {
|
||||
err = generation.reconcileExtractor(action, a, l, b, cs)
|
||||
}
|
||||
if err == nil {
|
||||
_, err = db.Exec(action, "UPDATE allocations SET state='evidence_held' WHERE id=$1", l.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
if err == nil {
|
||||
_, err = db.Exec(action, "UPDATE attempts SET phase='finished',outcome='interrupted',telemetry='partial',cleanup='evidence_held',error='Independent watchdog stopped expired attempt; evidence retained',finished_at=now(),lease_until=NULL WHERE id=$1 AND lease_owner=$2 AND lease_until>now()", id, generation.id)
|
||||
_, _ = db.Exec(action, "UPDATE runs SET status='failed' WHERE id=$1", a.RunID)
|
||||
_, _ = db.Exec(action, "UPDATE jobs SET status='failed',updated_at=now() WHERE id=$1", a.JobID)
|
||||
} else {
|
||||
blockers = append(blockers, "Watchdog could not establish owned VM stop: "+err.Error())
|
||||
}
|
||||
cs.close()
|
||||
cancel()
|
||||
}
|
||||
raw, _ := json.Marshal(blockers)
|
||||
_, err = db.Exec(ctx, "INSERT INTO heartbeats(name,seen_at,ready,blockers) VALUES('watchdog',now(),$1,$2) ON CONFLICT(name) DO UPDATE SET seen_at=now(),ready=excluded.ready,blockers=excluded.blockers", len(blockers) == 0, raw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-tick.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,332 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"net/url"
|
||||
"os"
|
||||
"otche/internal/pve"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type sourceSeal struct {
|
||||
State string `json:"state"`
|
||||
RevisionID string `json:"revision_id"`
|
||||
ConfigDigest string `json:"config_digest"`
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
SourceRef string `json:"source_ref"`
|
||||
SealedAt time.Time `json:"sealed_at"`
|
||||
}
|
||||
|
||||
func digest(v any) string {
|
||||
b, _ := json.Marshal(v)
|
||||
h := sha256.Sum256(b)
|
||||
return hex.EncodeToString(h[:])
|
||||
}
|
||||
func atomicJSON(path string, v any) error {
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = os.MkdirAll(filepath.Dir(path), 0700); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.CreateTemp(filepath.Dir(path), ".otche-")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
name := f.Name()
|
||||
defer os.Remove(name)
|
||||
if err = f.Chmod(0600); err == nil {
|
||||
_, err = f.Write(b)
|
||||
}
|
||||
if err == nil {
|
||||
err = f.Sync()
|
||||
}
|
||||
closeErr := f.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if closeErr != nil {
|
||||
return closeErr
|
||||
}
|
||||
return os.Rename(name, path)
|
||||
}
|
||||
func sourceConfig(ctx context.Context, c *pve.Client, s Source) (map[string]any, string, error) {
|
||||
state, err := c.Status(ctx, s.Node, s.VMID)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if state != "stopped" {
|
||||
return nil, "", errors.New("source must already be stopped; worker never stops a master")
|
||||
}
|
||||
cfg, err := c.Config(ctx, s.Node, s.VMID)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if pve.Text(cfg["template"]) != "" && pve.Text(cfg["template"]) != "0" {
|
||||
return nil, "", errors.New("source must be ordinary VM, template=0")
|
||||
}
|
||||
if err = c.NoPending(ctx, s.Node, s.VMID, ""); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if pve.Text(cfg["lock"]) != "" {
|
||||
return nil, "", errors.New("source has active PVE lock")
|
||||
}
|
||||
for k, v := range cfg {
|
||||
str := pve.Text(v)
|
||||
if strings.HasPrefix(k, "hostpci") || strings.HasPrefix(k, "usb") || k == "args" || k == "hookscript" || strings.HasPrefix(k, "virtiofs") || strings.HasPrefix(k, "unused") {
|
||||
return nil, "", errors.New("source contains unapproved passthrough, hook or unused disk")
|
||||
}
|
||||
if (k == s.CDSlot || diskHas(str, "media", "cdrom")) && !cdromMatches(str, "none") {
|
||||
return nil, "", errors.New("source has inserted or ambiguous CD media")
|
||||
}
|
||||
if strings.HasPrefix(k, "net") {
|
||||
bridge := ""
|
||||
for _, part := range strings.Split(str, ",") {
|
||||
if strings.HasPrefix(part, "bridge=") {
|
||||
bridge = strings.TrimPrefix(part, "bridge=")
|
||||
}
|
||||
}
|
||||
allowed := false
|
||||
for _, b := range s.AllowedBridges {
|
||||
if b == bridge {
|
||||
allowed = true
|
||||
}
|
||||
}
|
||||
if !allowed {
|
||||
return nil, "", errors.New("source inherited network is not an approved lab bridge")
|
||||
}
|
||||
}
|
||||
}
|
||||
var diskBytes int64
|
||||
for key, value := range cfg {
|
||||
if !cdromMatches(pve.Text(value), "none") && (strings.HasPrefix(key, "scsi") && key != "scsihw" || strings.HasPrefix(key, "sata") || strings.HasPrefix(key, "ide") || strings.HasPrefix(key, "virtio") || strings.HasPrefix(key, "efidisk") || strings.HasPrefix(key, "tpmstate")) {
|
||||
size, sizeErr := diskSize(pve.Text(value))
|
||||
if sizeErr != nil {
|
||||
return nil, "", errors.New("source disk size cannot be reserved safely")
|
||||
}
|
||||
diskBytes += size
|
||||
}
|
||||
}
|
||||
if diskBytes <= 0 || diskBytes > s.MaxDiskBytes {
|
||||
return nil, "", errors.New("source current disk capacity exceeds configured full-clone reservation")
|
||||
}
|
||||
if pve.Text(cfg["agent"]) == "" || pve.Text(cfg["agent"]) == "0" || pve.Text(cfg["vga"]) == "none" {
|
||||
return nil, "", errors.New("source requires QGA and graphical console")
|
||||
}
|
||||
delete(cfg, "digest")
|
||||
return cfg, digest(cfg), nil
|
||||
}
|
||||
func sourceLock(ctx context.Context, db *pgxpool.Pool, ref string) (*pgxpool.Conn, error) {
|
||||
bounded, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
conn, err := db.Acquire(bounded)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
_, err = conn.Exec(bounded, "SELECT pg_advisory_lock(hashtextextended($1,731))", ref)
|
||||
if err != nil {
|
||||
conn.Conn().Close(context.Background())
|
||||
conn.Release()
|
||||
return nil, err
|
||||
}
|
||||
return conn, nil
|
||||
}
|
||||
func unlockSource(conn *pgxpool.Conn, ref string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
_, err := conn.Exec(ctx, "SELECT pg_advisory_unlock(hashtextextended($1,731))", ref)
|
||||
if err != nil {
|
||||
conn.Conn().Close(ctx)
|
||||
}
|
||||
conn.Release()
|
||||
}
|
||||
|
||||
// ValidateSource records a stopped, immutable maintenance generation. It never
|
||||
// changes the VM. Disk content immutability depends on honoring BeginMaintenance;
|
||||
// a configuration hash cannot detect out-of-band writes inside a source disk.
|
||||
func ValidateSource(ctx context.Context, db *pgxpool.Pool, configPath, profileID, sourceRef string) (string, error) {
|
||||
cfg, err := loadConfig(configPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
s, ok := cfg.Sources[sourceRef]
|
||||
if !ok {
|
||||
return "", errors.New("source_ref is not configured")
|
||||
}
|
||||
conn, err := sourceLock(ctx, db, sourceRef)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer unlockSource(conn, sourceRef)
|
||||
var state string
|
||||
if err = conn.QueryRow(ctx, "SELECT state FROM profiles WHERE id=$1", profileID).Scan(&state); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if state != "maintenance" {
|
||||
return "", errors.New("profile must be in maintenance before source validation")
|
||||
}
|
||||
if err = drained(ctx, db, sourceRef); err != nil {
|
||||
return "", err
|
||||
}
|
||||
b := cfg.Owners[s.OwnerID]
|
||||
cs, err := b.clients()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer cs.close()
|
||||
_, dg, err := sourceConfig(ctx, cs.provisioner, s)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
id := newID()
|
||||
seal := sourceSeal{State: "frozen", RevisionID: id, ConfigDigest: dg, Fingerprint: digest([]string{sourceRef, id, dg}), SourceRef: sourceRef, SealedAt: time.Now().UTC()}
|
||||
tx, err := conn.Begin(ctx)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
_, err = tx.Exec(ctx, "INSERT INTO revisions(id,profile_id,source_ref,fingerprint,config_digest,qualification) VALUES($1,$2,$3,$4,$5,$6)", id, profileID, sourceRef, seal.Fingerprint, dg, []byte(`{"worker_validated":true,"state":"unqualified"}`))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, err = tx.Exec(ctx, "UPDATE profiles SET current_revision_id=$2,state='maintenance',qualification='unqualified',enabled=false,online_available=false,reason='Worker validated stopped source; qualification required' WHERE id=$1", profileID, id)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = atomicJSON(s.SealPath, seal); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
func drained(ctx context.Context, db *pgxpool.Pool, ref string) error {
|
||||
var n int
|
||||
err := db.QueryRow(ctx, `SELECT count(*) FROM attempts a JOIN runs r ON r.id=a.run_id JOIN revisions v ON v.id=r.revision_id LEFT JOIN allocations l ON l.attempt_id=a.id WHERE v.source_ref=$1 AND a.phase<>'finished' AND (l.id IS NULL OR l.state IN ('reserved','clone_intent','cloning','uncertain'))`, ref).Scan(&n)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if n != 0 {
|
||||
return errors.New("source not drained: accepted jobs still need current-disk copies")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// BeginMaintenance closes admission and refuses to release the immutable seal
|
||||
// until every accepted old-revision copy has materialized or been cancelled.
|
||||
func BeginMaintenance(ctx context.Context, db *pgxpool.Pool, configPath, sourceRef string) error {
|
||||
cfg, err := loadConfig(configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s, ok := cfg.Sources[sourceRef]
|
||||
if !ok {
|
||||
return errors.New("source_ref is not configured")
|
||||
}
|
||||
conn, err := sourceLock(ctx, db, sourceRef)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer unlockSource(conn, sourceRef)
|
||||
_, err = conn.Exec(ctx, `UPDATE profiles SET state='maintenance',enabled=false,reason='Maintenance requested; waiting for source drain' WHERE id IN (SELECT profile_id FROM revisions WHERE source_ref=$1)`, sourceRef)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = drained(ctx, db, sourceRef); err != nil {
|
||||
return err
|
||||
}
|
||||
return atomicJSON(s.SealPath, sourceSeal{State: "maintenance", SourceRef: sourceRef})
|
||||
}
|
||||
func checkSeal(s Source, ref, revision, expected string) error {
|
||||
var seal sourceSeal
|
||||
b, err := os.ReadFile(s.SealPath)
|
||||
if err != nil || json.Unmarshal(b, &seal) != nil {
|
||||
return errors.New("source immutable maintenance seal unavailable")
|
||||
}
|
||||
if seal.State != "frozen" || seal.SourceRef != ref || seal.RevisionID != revision || seal.ConfigDigest != expected {
|
||||
return errors.New("source revision seal is not frozen or differs from accepted revision")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func SyncBindings(ctx context.Context, db *pgxpool.Pool, configPath string) error {
|
||||
cfg, err := loadConfig(configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = db.Exec(ctx, "UPDATE bindings SET configured=false,reason='Binding requires worker validation'"); err != nil {
|
||||
return err
|
||||
}
|
||||
for owner, b := range cfg.Owners {
|
||||
reason := ""
|
||||
var expiresAt *time.Time
|
||||
cs, e := b.clients()
|
||||
if e != nil {
|
||||
reason = e.Error()
|
||||
} else {
|
||||
e = cs.authenticate(ctx)
|
||||
if e == nil {
|
||||
var proof IsolationProof
|
||||
proof, e = b.isolationProof(owner)
|
||||
if e == nil {
|
||||
expiresAt = &proof.ExpiresAt
|
||||
}
|
||||
}
|
||||
if e == nil {
|
||||
var pool map[string]any
|
||||
e = cs.provisioner.Do(ctx, "GET", "/pools/"+b.Pool, nil, &pool)
|
||||
if e == nil {
|
||||
e = storageHeadroom(ctx, cs.provisioner, b, 0)
|
||||
}
|
||||
}
|
||||
cs.close()
|
||||
if e != nil {
|
||||
reason = e.Error()
|
||||
}
|
||||
}
|
||||
_, err = db.Exec(ctx, `INSERT INTO bindings(owner_id,pool,iso_storage,disk_storage,node,configured,reason,isolation_expires_at) VALUES($1,$2,$3,$4,$5,$6,$7,$8) ON CONFLICT(owner_id) DO UPDATE SET pool=excluded.pool,iso_storage=excluded.iso_storage,disk_storage=excluded.disk_storage,node=excluded.node,configured=excluded.configured,reason=excluded.reason,isolation_expires_at=excluded.isolation_expires_at`, owner, b.Pool, b.ISOStorage, b.DiskStorage, b.Node, reason == "", reason, expiresAt)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func storageHeadroom(ctx context.Context, c *pve.Client, b Binding, diskBytes int64) error {
|
||||
for _, storage := range []string{b.DiskStorage, b.ISOStorage} {
|
||||
var s struct {
|
||||
Avail int64 `json:"avail"`
|
||||
Active pve.Bool `json:"active"`
|
||||
}
|
||||
if err := c.Do(ctx, "GET", "/nodes/"+b.Node+"/storage/"+storage+"/status", nil, &s); err != nil {
|
||||
return err
|
||||
}
|
||||
need := b.MinStorageFreeBytes
|
||||
if storage == b.DiskStorage {
|
||||
need += diskBytes
|
||||
}
|
||||
if !s.Active || s.Avail < need {
|
||||
return fmt.Errorf("storage headroom insufficient for %s", storage)
|
||||
}
|
||||
}
|
||||
var node struct {
|
||||
Memory struct {
|
||||
Free int64 `json:"free"`
|
||||
} `json:"memory"`
|
||||
}
|
||||
if err := c.Do(ctx, "GET", "/nodes/"+b.Node+"/status", url.Values{}, &node); err != nil {
|
||||
return err
|
||||
}
|
||||
if node.Memory.Free < b.MinMemoryFreeBytes {
|
||||
return errors.New("node memory headroom insufficient")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,268 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"otche/internal/pve"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type engine struct {
|
||||
db *pgxpool.Pool
|
||||
cfg Config
|
||||
root, id string
|
||||
}
|
||||
type attempt struct {
|
||||
ID, RunID, JobID, OwnerID, CommandID, ProfileID, RevisionID, SourceRef, ConfigDigest, Phase, Filename, SHA256, StorageKey, Lease string
|
||||
Settings json.RawMessage
|
||||
Qualification json.RawMessage
|
||||
Size int64
|
||||
StartedAt, DeadlineAt *time.Time
|
||||
}
|
||||
type settings struct {
|
||||
Internet string `json:"internet"`
|
||||
Duration int `json:"duration_seconds"`
|
||||
Privilege string `json:"privilege"`
|
||||
GrubPaths []string `json:"grub_paths,omitempty"`
|
||||
}
|
||||
type allocation struct {
|
||||
ID string
|
||||
VMID int
|
||||
State string
|
||||
UPID *string
|
||||
Metadata json.RawMessage
|
||||
}
|
||||
|
||||
func Run(ctx context.Context, db *pgxpool.Pool, artifactRoot, configPath string) error {
|
||||
cfg, err := loadConfig(configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
e := &engine{db: db, cfg: cfg, root: artifactRoot, id: newID()}
|
||||
if err = SyncBindings(ctx, db, configPath); err != nil {
|
||||
return err
|
||||
}
|
||||
var wg sync.WaitGroup
|
||||
defer wg.Wait()
|
||||
slots := make(chan struct{}, cfg.Concurrent)
|
||||
tick := time.NewTicker(2 * time.Second)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
blockers := []string{}
|
||||
if _, err = db.Exec(ctx, "UPDATE bindings SET configured=false,reason='Isolation proof expired or requires revalidation' WHERE configured AND NOT COALESCE(isolation_expires_at>now(),false)"); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(cfg.Owners) == 0 {
|
||||
blockers = append(blockers, "No file-backed owner bindings configured")
|
||||
}
|
||||
if len(cfg.Sources) == 0 {
|
||||
blockers = append(blockers, "No worker-validated source mappings configured")
|
||||
}
|
||||
if cfg.WatchdogRequired {
|
||||
var ok bool
|
||||
err = db.QueryRow(ctx, "SELECT EXISTS(SELECT 1 FROM heartbeats WHERE name='watchdog' AND ready AND seen_at>now()-interval '20 seconds')").Scan(&ok)
|
||||
if err != nil || !ok {
|
||||
blockers = append(blockers, "Independent watchdog heartbeat missing")
|
||||
}
|
||||
}
|
||||
raw, _ := json.Marshal(blockers)
|
||||
_, err = db.Exec(ctx, "INSERT INTO heartbeats(name,seen_at,ready,blockers) VALUES('worker',now(),$1,$2) ON CONFLICT(name) DO UPDATE SET seen_at=now(),ready=excluded.ready,blockers=excluded.blockers", len(blockers) == 0, raw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(blockers) == 0 {
|
||||
select {
|
||||
case slots <- struct{}{}:
|
||||
a, claimErr := e.claim(ctx)
|
||||
if claimErr != nil {
|
||||
<-slots
|
||||
if !errors.Is(claimErr, pgx.ErrNoRows) {
|
||||
return claimErr
|
||||
}
|
||||
} else {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
defer func() { <-slots }()
|
||||
claimed := *e
|
||||
claimed.id = a.Lease
|
||||
claimed.runClaim(ctx, a)
|
||||
}()
|
||||
}
|
||||
default:
|
||||
}
|
||||
}
|
||||
if err = e.releaseEvidence(ctx); err != nil {
|
||||
e.healthError(ctx, "Evidence release requires operator attention")
|
||||
}
|
||||
if len(blockers) == 0 {
|
||||
if err = e.qualify(ctx); err != nil {
|
||||
e.healthError(ctx, err.Error())
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-tick.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
func (e *engine) healthError(ctx context.Context, message string) {
|
||||
_, _ = e.db.Exec(ctx, "INSERT INTO events(kind,message) VALUES('worker', $1)", message)
|
||||
}
|
||||
func (e *engine) claim(ctx context.Context) (attempt, error) {
|
||||
tx, err := e.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return attempt{}, err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
if _, err = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(731001)"); err != nil {
|
||||
return attempt{}, err
|
||||
}
|
||||
var active int
|
||||
if err = tx.QueryRow(ctx, "SELECT count(*) FROM attempts WHERE phase<>'finished' AND lease_until>now()").Scan(&active); err != nil {
|
||||
return attempt{}, err
|
||||
}
|
||||
if active >= e.cfg.Concurrent {
|
||||
return attempt{}, pgx.ErrNoRows
|
||||
}
|
||||
var id string
|
||||
err = tx.QueryRow(ctx, `SELECT a.id FROM attempts a JOIN runs r ON r.id=a.run_id JOIN jobs j ON j.id=r.job_id WHERE a.phase<>'finished' AND (a.lease_until IS NULL OR a.lease_until<now()) AND NOT EXISTS(SELECT 1 FROM attempts b JOIN runs br ON br.id=b.run_id JOIN jobs bj ON bj.id=br.job_id WHERE bj.owner_id=j.owner_id AND b.lease_until>now() AND b.phase<>'finished') ORDER BY a.created_at FOR UPDATE OF a SKIP LOCKED LIMIT 1`).Scan(&id)
|
||||
if err != nil {
|
||||
return attempt{}, err
|
||||
}
|
||||
generation := newID()
|
||||
_, err = tx.Exec(ctx, "UPDATE attempts SET lease_owner=$2,lease_until=now()+interval '30 seconds' WHERE id=$1", id, generation)
|
||||
if err != nil {
|
||||
return attempt{}, err
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return attempt{}, err
|
||||
}
|
||||
a, err := e.load(ctx, id)
|
||||
a.Lease = generation
|
||||
return a, err
|
||||
}
|
||||
func (e *engine) load(ctx context.Context, id string) (attempt, error) {
|
||||
var a attempt
|
||||
err := e.db.QueryRow(ctx, `SELECT a.id,a.run_id,r.job_id,j.owner_id,a.command_id,r.profile_id,r.revision_id,v.source_ref,v.config_digest,a.phase,j.execution_filename,u.sha256,u.storage_key,j.settings,COALESCE(v.qualification,'{}'),u.size,a.started_at,a.deadline_at FROM attempts a JOIN runs r ON r.id=a.run_id JOIN jobs j ON j.id=r.job_id JOIN uploads u ON u.id=j.upload_id JOIN revisions v ON v.id=r.revision_id WHERE a.id=$1`, id).Scan(&a.ID, &a.RunID, &a.JobID, &a.OwnerID, &a.CommandID, &a.ProfileID, &a.RevisionID, &a.SourceRef, &a.ConfigDigest, &a.Phase, &a.Filename, &a.SHA256, &a.StorageKey, &a.Settings, &a.Qualification, &a.Size, &a.StartedAt, &a.DeadlineAt)
|
||||
return a, err
|
||||
}
|
||||
func (e *engine) fence(ctx context.Context, a attempt) error {
|
||||
var ok, cancel bool
|
||||
err := e.db.QueryRow(ctx, "SELECT a.lease_owner=$2 AND a.lease_until>now(),j.cancel_requested FROM attempts a JOIN runs r ON r.id=a.run_id JOIN jobs j ON j.id=r.job_id WHERE a.id=$1", a.ID, e.id).Scan(&ok, &cancel)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !ok {
|
||||
return errors.New("attempt lease lost")
|
||||
}
|
||||
if cancel {
|
||||
return errors.New("cancelled")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (e *engine) phase(ctx context.Context, a attempt, phase string) error {
|
||||
tag, err := e.db.Exec(ctx, "UPDATE attempts SET phase=$3 WHERE id=$1 AND lease_owner=$2 AND lease_until>now()", a.ID, e.id, phase)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if tag.RowsAffected() != 1 {
|
||||
return errors.New("attempt lease lost")
|
||||
}
|
||||
_, err = e.db.Exec(ctx, "INSERT INTO events(job_id,attempt_id,kind,message) VALUES($1,$2,'phase',$3)", a.JobID, a.ID, phase)
|
||||
return err
|
||||
}
|
||||
func (e *engine) runClaim(parent context.Context, a attempt) {
|
||||
ctx, cancel := context.WithCancel(parent)
|
||||
defer cancel()
|
||||
done := make(chan struct{})
|
||||
defer close(done)
|
||||
go func() {
|
||||
t := time.NewTicker(5 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
return
|
||||
case <-t.C:
|
||||
renew, finish := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
tag, err := e.db.Exec(renew, "UPDATE attempts SET lease_until=now()+interval '30 seconds' WHERE id=$1 AND lease_owner=$2 AND lease_until>now()", a.ID, e.id)
|
||||
if err != nil || tag.RowsAffected() != 1 {
|
||||
finish()
|
||||
cancel()
|
||||
return
|
||||
}
|
||||
if err = e.fence(renew, a); err != nil {
|
||||
cancel()
|
||||
}
|
||||
finish()
|
||||
}
|
||||
}
|
||||
}()
|
||||
_, _ = e.db.Exec(ctx, "UPDATE jobs SET status='running',updated_at=now() WHERE id=$1", a.JobID)
|
||||
_, _ = e.db.Exec(ctx, "UPDATE runs SET status='running' WHERE id=$1", a.RunID)
|
||||
outcome, findings, telemetry, cleanup, report, err := e.execute(ctx, a)
|
||||
final, c := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer c()
|
||||
msg := ""
|
||||
if err != nil {
|
||||
msg = err.Error()
|
||||
}
|
||||
if len(msg) > 1500 {
|
||||
msg = msg[:1500]
|
||||
}
|
||||
tag, finishErr := e.db.Exec(final, `UPDATE attempts SET phase='finished',outcome=$3,findings=$4,telemetry=$5,cleanup=$6,error=$7,report=$8,finished_at=now(),lease_until=NULL WHERE id=$1 AND lease_owner=$2 AND lease_until>now()`, a.ID, e.id, outcome, findings, telemetry, cleanup, msg, report)
|
||||
if finishErr != nil || tag.RowsAffected() == 0 {
|
||||
return
|
||||
}
|
||||
status := "completed"
|
||||
if outcome == "cancelled" {
|
||||
status = "cancelled"
|
||||
} else if outcome == "error" || outcome == "interrupted" || outcome == "delivery_error" {
|
||||
status = "failed"
|
||||
}
|
||||
_, _ = e.db.Exec(final, "UPDATE runs SET status=$2 WHERE id=$1", a.RunID, status)
|
||||
_, _ = e.db.Exec(final, `UPDATE jobs SET status=CASE WHEN EXISTS(SELECT 1 FROM runs WHERE job_id=$1 AND status IN ('queued','running')) THEN 'running' WHEN cancel_requested THEN 'cancelled' WHEN EXISTS(SELECT 1 FROM runs WHERE job_id=$1 AND status='failed') THEN 'failed' ELSE 'completed' END,updated_at=now() WHERE id=$1`, a.JobID)
|
||||
}
|
||||
func (e *engine) getAllocation(ctx context.Context, a attempt) (allocation, error) {
|
||||
var l allocation
|
||||
var owner, node, pool string
|
||||
err := e.db.QueryRow(ctx, "SELECT id,vmid,state,upid,metadata,owner_id,node,pool FROM allocations WHERE attempt_id=$1", a.ID).Scan(&l.ID, &l.VMID, &l.State, &l.UPID, &l.Metadata, &owner, &node, &pool)
|
||||
if err != nil {
|
||||
return l, err
|
||||
}
|
||||
b, ok := e.cfg.Owners[a.OwnerID]
|
||||
if !ok || owner != a.OwnerID || node != b.Node || pool != b.Pool {
|
||||
return l, errors.New("recorded allocation owner/node/pool differs from configured binding")
|
||||
}
|
||||
return l, nil
|
||||
}
|
||||
func (e *engine) own(a attempt, l allocation, b Binding) pve.Ownership {
|
||||
return pve.Ownership{Node: b.Node, VMID: l.VMID, Pool: b.Pool, OwnerID: a.OwnerID, AttemptID: a.ID, Name: "otche-" + a.ID, Protected: e.cfg.protected()}
|
||||
}
|
||||
func (e *engine) recordAction(ctx context.Context, a attempt, l allocation, state, upid string) error {
|
||||
if err := e.fence(ctx, a); err != nil {
|
||||
return err
|
||||
}
|
||||
tag, err := e.db.Exec(ctx, "UPDATE allocations SET state=$2,upid=NULLIF($3,'') WHERE id=$1 AND EXISTS(SELECT 1 FROM attempts WHERE id=$4 AND lease_owner=$5 AND lease_until>now())", l.ID, state, upid, a.ID, e.id)
|
||||
if err == nil && tag.RowsAffected() != 1 {
|
||||
return errors.New("attempt lease lost")
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (e *engine) deadline(ctx context.Context, a attempt, t time.Time) error {
|
||||
tag, err := e.db.Exec(ctx, "UPDATE attempts SET deadline_at=$3 WHERE id=$1 AND lease_owner=$2", a.ID, e.id, t)
|
||||
if err == nil && tag.RowsAffected() != 1 {
|
||||
return fmt.Errorf("attempt lease lost")
|
||||
}
|
||||
return err
|
||||
}
|
||||
Executable
+15
@@ -0,0 +1,15 @@
|
||||
#!/bin/sh
|
||||
# Run only in a NEW clean Debian Linux extractor source VM, not PVE or CT7000.
|
||||
# Install signed/trusted Debian packages during clean source maintenance first:
|
||||
# apt-get install python3 ntfs-3g qemu-guest-agent util-linux
|
||||
set -eu
|
||||
[ "$(id -u)" = 0 ] || { echo 'Run in extractor VM as root' >&2; exit 1; }
|
||||
[ "${1:-}" = '--confirm-disposable-extractor-source' ] || { echo 'Explicit extractor-source confirmation required' >&2; exit 1; }
|
||||
command -v python3 >/dev/null
|
||||
command -v ntfs-3g >/dev/null
|
||||
command -v lsblk >/dev/null
|
||||
# This is intentionally not an automated disk formatter or host configuration tool.
|
||||
install -d -m 0700 /var/lib/otche/control /var/lib/otche/export /var/lib/otche/mount
|
||||
install -o root -g root -m 0755 "$(dirname "$0")/otche-extract" /usr/local/sbin/otche-extract
|
||||
systemctl enable --now qemu-guest-agent
|
||||
printf '%s\n' 'Collector installed. Power off clean source; remove all NICs/passthrough in PVE; qualify read-only reassignment on disposable probes before setting extractor proof.'
|
||||
Executable
+153
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Trusted PVE administrator onboarding. Run on a PVE node with pveum, not in the API.
|
||||
Creates one owner's scoped tokens, using separate users or one clean existing PVE user.
|
||||
Does not create storage, modify master, start VMs, change firewall or grant broad ACLs.
|
||||
Existing resources are refused; --service-user permits only an enabled unprivileged user.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def run(*args):
|
||||
result = subprocess.run(args, check=True, capture_output=True, text=True, timeout=60)
|
||||
return result.stdout
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument('--owner', required=True, help='Panel owner UUID')
|
||||
p.add_argument('--source-vmid', type=int, action='append', required=True, help='Repeat for each permitted ordinary Windows source')
|
||||
p.add_argument('--extractor-source-vmid', type=int, help='Optional stopped ordinary no-network Linux source')
|
||||
p.add_argument('--node', required=True)
|
||||
p.add_argument('--pool', required=True)
|
||||
p.add_argument('--iso-storage', required=True)
|
||||
p.add_argument('--disk-storage', required=True)
|
||||
p.add_argument('--approved-bridge', help='Optional dedicated sandbox bridge, never production vmbr0')
|
||||
p.add_argument('--service-user', help='Existing enabled, empty ASCII pve-realm account; all five tokens use privsep=1')
|
||||
p.add_argument('--output-dir', type=Path, required=True)
|
||||
p.add_argument('--apply', action='store_true')
|
||||
a = p.parse_args()
|
||||
if not re.fullmatch(r'[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}', a.owner):
|
||||
p.error('canonical owner UUID required')
|
||||
for value in (a.node, a.pool, a.iso_storage, a.disk_storage):
|
||||
if not re.fullmatch(r'[a-zA-Z0-9][a-zA-Z0-9_.-]*', value):
|
||||
p.error('invalid PVE identifier')
|
||||
if any(vmid < 100 or vmid == 7000 for vmid in a.source_vmid):
|
||||
p.error('invalid source VM')
|
||||
if a.extractor_source_vmid is not None and (a.extractor_source_vmid < 100 or a.extractor_source_vmid in (7000,7001) or a.extractor_source_vmid in a.source_vmid):
|
||||
p.error('extractor source must be distinct from protected Windows/control IDs')
|
||||
if a.iso_storage == 'local' or a.disk_storage == 'tank-store':
|
||||
p.error('dedicated private ISO and clone disk storage required; lab/control storages forbidden')
|
||||
if a.approved_bridge == 'vmbr0':
|
||||
p.error('production vmbr0 is not a sandbox')
|
||||
if a.service_user and not re.fullmatch(r'[A-Za-z][A-Za-z0-9_.-]*@pve', a.service_user):
|
||||
p.error('service user must be an ASCII pve-realm account')
|
||||
if a.iso_storage == a.disk_storage:
|
||||
p.error('ISO and disk storage must be distinct')
|
||||
prefix = 'otche-' + a.owner.replace('-', '')[:12]
|
||||
rules = {
|
||||
'provisioner': {
|
||||
**{'/vms/' + str(vmid): ('Source', 'VM.Audit VM.Clone') for vmid in set(a.source_vmid)},
|
||||
'/pool/' + a.pool: ('Provision', 'VM.Allocate VM.Audit VM.PowerMgmt VM.Config.CDROM VM.Config.Network VM.Config.Options VM.Config.Disk VM.Config.HWType'),
|
||||
'/pool/' + a.pool + ':audit': ('PoolAudit', 'Pool.Audit'),
|
||||
'/storage/' + a.disk_storage: ('Disk', 'Datastore.AllocateSpace Datastore.Audit'),
|
||||
'/storage/' + a.iso_storage: ('ISOAudit', 'Datastore.Audit'),
|
||||
'/nodes/' + a.node: ('Node', 'Sys.Audit'),
|
||||
},
|
||||
'runtime': {'/pool/' + a.pool: ('Runtime', 'VM.Audit VM.GuestAgent.Unrestricted')},
|
||||
'recorder': {'/pool/' + a.pool: ('Recorder', 'VM.Audit VM.Console')},
|
||||
'uploader': {'/storage/' + a.iso_storage: ('Uploader', 'Datastore.Audit Datastore.AllocateTemplate')},
|
||||
'housekeeping': {'/storage/' + a.iso_storage: ('Housekeeping', 'Datastore.Audit Datastore.Allocate')},
|
||||
}
|
||||
if a.extractor_source_vmid is not None:
|
||||
rules['provisioner']['/vms/' + str(a.extractor_source_vmid)] = ('ExtractorSource', 'VM.Audit VM.Clone')
|
||||
if a.approved_bridge:
|
||||
rules['provisioner']['/sdn/zones/localnetwork/' + a.approved_bridge] = ('Network', 'SDN.Use')
|
||||
if not a.apply:
|
||||
print(json.dumps({'pool': a.pool, 'owner': a.owner, 'service_user': a.service_user,
|
||||
'token_privsep': 1, 'parent_scope': 'union of listed token grants only', 'grants': rules,
|
||||
'prerequisites': ['distinct nonalias private storage already provisioned',
|
||||
'live disposable cross-owner allow/deny probes',
|
||||
'worker-only read-only credential mounts',
|
||||
'separate watchdog service',
|
||||
'signed Windows runner policy and active console session',
|
||||
'qualification before publication']}, indent=2))
|
||||
return
|
||||
run('pveversion', '--verbose')
|
||||
cluster = json.loads(run('pvesh', 'get', '/cluster/status', '--output-format', 'json'))
|
||||
if not any(item.get('type') == 'cluster' and item.get('quorate') == 1 for item in cluster):
|
||||
p.error('a quorate cluster is required')
|
||||
users = json.loads(run('pveum', 'user', 'list', '--full', '1', '--output-format', 'json'))
|
||||
pools = json.loads(run('pveum', 'pool', 'list', '--output-format', 'json'))
|
||||
roles = json.loads(run('pveum', 'role', 'list', '--output-format', 'json'))
|
||||
if any(item['poolid'] == a.pool for item in pools):
|
||||
p.error('pool already exists; refusing partial reapply')
|
||||
planned_roles = {prefix + '-' + purpose + '-' + str(index)
|
||||
for purpose, grants in rules.items() for index in range(len(grants))}
|
||||
if any(item['roleid'] in planned_roles for item in roles):
|
||||
p.error('planned role already exists; refusing partial reapply')
|
||||
if a.service_user:
|
||||
matches = [item for item in users if item['userid'] == a.service_user]
|
||||
if len(matches) != 1 or matches[0].get('enable') != 1 or matches[0].get('expire', 0) != 0:
|
||||
p.error('service user must already exist, enabled and non-expiring')
|
||||
if matches[0].get('groups') or matches[0].get('tokens'):
|
||||
p.error('service user must have no groups or tokens')
|
||||
permissions = json.loads(run('pveum', 'user', 'permissions', a.service_user, '--output-format', 'json'))
|
||||
acl = json.loads(run('pveum', 'acl', 'list', '--output-format', 'json'))
|
||||
if any(permissions.values()) or any(item.get('ugid') == a.service_user for item in acl):
|
||||
p.error('service user must have no effective privileges or ACL entries')
|
||||
elif any(item['userid'] in {prefix + '-' + purpose + '@pve' for purpose in rules} for item in users):
|
||||
p.error('planned user already exists; refusing partial reapply')
|
||||
inventory = json.loads(run('pvesh', 'get', '/storage', '--output-format', 'json'))
|
||||
selected = {item['storage']: item for item in inventory if item['storage'] in (a.iso_storage, a.disk_storage)}
|
||||
if len(selected) != 2:
|
||||
p.error('both private storages must already exist')
|
||||
for storage, content in ((a.iso_storage, 'iso'), (a.disk_storage, 'images')):
|
||||
item = selected[storage]
|
||||
if content not in item.get('content', '').split(',') or item.get('disable'):
|
||||
p.error('private storage content is unavailable')
|
||||
if item.get('nodes') and a.node not in item['nodes'].split(','):
|
||||
p.error('private storage is unavailable on selected node')
|
||||
if item.get('type') == 'dir':
|
||||
path = Path(item['path']).resolve(strict=True)
|
||||
for other in inventory:
|
||||
if other['storage'] != storage and other.get('type') == 'dir':
|
||||
other_path = Path(other['path']).resolve()
|
||||
if path == other_path or path in other_path.parents or other_path in path.parents:
|
||||
p.error('private storage overlaps another storage path')
|
||||
elif item.get('type') == 'zfspool':
|
||||
if any(other['storage'] != storage and other.get('type') == 'zfspool' and other.get('pool') == item.get('pool') for other in inventory):
|
||||
p.error('private storage aliases another ZFS storage')
|
||||
else:
|
||||
p.error('private storage requires independently provisioned dir or ZFS backing')
|
||||
run('pvesh', 'get', '/nodes/' + a.node + '/storage/' + storage + '/status', '--output-format', 'json')
|
||||
os.umask(0o077)
|
||||
a.output_dir.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
run('pveum', 'pool', 'add', a.pool, '--comment', 'otche owner ' + a.owner)
|
||||
for purpose, grants in rules.items():
|
||||
user = a.service_user or prefix + '-' + purpose + '@pve'
|
||||
if not a.service_user:
|
||||
run('pveum', 'user', 'add', user, '--comment', 'otche ' + purpose + ' owner ' + a.owner)
|
||||
token_name = prefix + '-' + purpose if a.service_user else 'worker'
|
||||
token = json.loads(run('pveum', 'user', 'token', 'add', user, token_name, '--privsep', '1', '--output-format', 'json'))
|
||||
credential = {'token_id': token['full-tokenid'], 'secret': token['value']}
|
||||
with (a.output_dir / (purpose + '.json')).open('x', encoding='utf-8') as f:
|
||||
json.dump(credential, f)
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
for index, (path, (label, privileges)) in enumerate(grants.items()):
|
||||
role = prefix + '-' + purpose + '-' + str(index)
|
||||
run('pveum', 'role', 'add', role, '--privs', privileges)
|
||||
path = path.removesuffix(':audit')
|
||||
run('pveum', 'acl', 'modify', path, '--users', user, '--roles', role, '--propagate', '1')
|
||||
run('pveum', 'acl', 'modify', path, '--tokens', credential['token_id'], '--roles', role, '--propagate', '1')
|
||||
print(purpose + ': ' + str(a.output_dir / (purpose + '.json')))
|
||||
print('Created scoped credentials. Not qualified: run real disposable isolation controls and source validation before enabling binding.')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Executable
+150
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/python3
|
||||
"""Installed only inside an isolated disposable Linux extractor VM.
|
||||
Never run on PVE, the control CT, or the Windows source. No repair or write mount.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def command(argv, timeout=30):
|
||||
return subprocess.run(argv, check=True, text=True, capture_output=True, timeout=timeout).stdout
|
||||
|
||||
|
||||
def atomic_json(path, value):
|
||||
temp = path.with_suffix('.tmp')
|
||||
with temp.open('x', encoding='utf-8') as handle:
|
||||
json.dump(value, handle, ensure_ascii=True)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(temp, path)
|
||||
|
||||
|
||||
def exact_directory(root, parts):
|
||||
current = root
|
||||
for part in parts:
|
||||
current = current / part
|
||||
if current.is_symlink() or not current.is_dir():
|
||||
raise RuntimeError('expected evidence directory missing or symlink')
|
||||
return current
|
||||
|
||||
|
||||
def main():
|
||||
if os.geteuid() != 0 or len(sys.argv) != 2:
|
||||
raise RuntimeError('fixed extractor requires root QGA control and one manifest')
|
||||
manifest_path = Path(sys.argv[1])
|
||||
if manifest_path.parent != Path('/var/lib/otche/control') or not re.fullmatch(r'[0-9a-f-]{36}\.json', manifest_path.name):
|
||||
raise RuntimeError('manifest path outside fixed control directory')
|
||||
if manifest_path.stat().st_size > 40960:
|
||||
raise RuntimeError('manifest too large')
|
||||
spec = json.loads(manifest_path.read_text(encoding='utf-8'))
|
||||
cid = spec['command_id']
|
||||
if not re.fullmatch(r'[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}', cid) or manifest_path.stem != cid:
|
||||
raise RuntimeError('invalid command identity')
|
||||
maximum = int(spec['max_bytes'])
|
||||
if not 1 <= maximum <= 1073741824 or not re.fullmatch(r'otche[0-9a-f]{16}', spec['serial']):
|
||||
raise RuntimeError('invalid collection limits or disk identity')
|
||||
target = Path('/var/lib/otche/export') / cid
|
||||
target.mkdir(mode=0o700, parents=False, exist_ok=False) # durable acceptance; never repeat
|
||||
report = {'command_id': cid, 'complete': False, 'errors': [], 'files': []}
|
||||
atomic_json(target / 'receipt.json', {'command_id': cid, 'state': 'accepted'})
|
||||
mount = Path('/var/lib/otche/mount') / cid
|
||||
mount.mkdir(mode=0o700, parents=False, exist_ok=False)
|
||||
mounted = False
|
||||
try:
|
||||
listing = json.loads(command(['lsblk', '-J', '-b', '-o', 'PATH,TYPE,SERIAL,SIZE,RO,FSTYPE']))
|
||||
matching = [x for x in listing['blockdevices'] if x.get('type') == 'disk' and str(x.get('serial', '')).strip() == spec['serial']]
|
||||
if len(matching) != 1:
|
||||
raise RuntimeError('allocated evidence serial is not unique')
|
||||
disk = matching[0]
|
||||
if int(disk['size']) != int(spec['disk_bytes']) or str(disk['ro']).lower() not in ('1', 'true'):
|
||||
raise RuntimeError('evidence size or hypervisor read-only property mismatch')
|
||||
if command(['blockdev', '--getro', disk['path']]).strip() != '1':
|
||||
raise RuntimeError('kernel does not report block device read-only')
|
||||
partitions = [disk] + disk.get('children', [])
|
||||
found = False
|
||||
copied = 0
|
||||
for partition in partitions:
|
||||
if str(partition.get('fstype', '')).lower() != 'ntfs':
|
||||
continue
|
||||
if str(partition.get('ro')).lower() not in ('1', 'true'):
|
||||
raise RuntimeError('partition is not read-only')
|
||||
try:
|
||||
command(['ntfs-3g', '-o', 'ro,norecover,noexec,nodev,nosuid', partition['path'], str(mount)])
|
||||
mounted = True
|
||||
try:
|
||||
folder = exact_directory(mount, ['ProgramData', 'Otche', 'results', cid])
|
||||
except RuntimeError:
|
||||
command(['umount', str(mount)])
|
||||
mounted = False
|
||||
continue
|
||||
found = True
|
||||
candidates = []
|
||||
with os.scandir(folder) as files:
|
||||
for entry in files:
|
||||
if entry.is_file(follow_symlinks=False):
|
||||
candidates.append(Path(entry.path))
|
||||
if spec.get('include_crash_dump'):
|
||||
windows = exact_directory(mount, ['Windows'])
|
||||
dump = windows / 'MEMORY.DMP'
|
||||
if dump.is_file() and not dump.is_symlink():
|
||||
candidates.append(dump)
|
||||
else:
|
||||
report['errors'].append('Requested system crash dump unavailable')
|
||||
if len(candidates) > 64:
|
||||
raise RuntimeError('artifact count exceeds bound')
|
||||
for index, source in enumerate(sorted(candidates)):
|
||||
name = re.sub(r'[^A-Za-z0-9_.-]', '_', source.name)[:120]
|
||||
output_name = f'{index:03d}-{name}'
|
||||
fd = os.open(source, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||
try:
|
||||
size = os.fstat(fd).st_size
|
||||
if not stat.S_ISREG(os.fstat(fd).st_mode):
|
||||
raise RuntimeError('evidence is not a regular file')
|
||||
if copied + size > maximum:
|
||||
report['errors'].append('Artifact byte budget exceeded: ' + name)
|
||||
continue
|
||||
with os.fdopen(fd, 'rb', closefd=False) as src, (target / output_name).open('xb') as dst:
|
||||
remaining = size
|
||||
while remaining:
|
||||
data = src.read(min(262144, remaining))
|
||||
if not data:
|
||||
raise RuntimeError('evidence file short read')
|
||||
dst.write(data)
|
||||
remaining -= len(data)
|
||||
dst.flush()
|
||||
os.fsync(dst.fileno())
|
||||
copied += size
|
||||
report['files'].append({'name': output_name, 'size': size})
|
||||
finally:
|
||||
os.close(fd)
|
||||
command(['umount', str(mount)])
|
||||
mounted = False
|
||||
break
|
||||
finally:
|
||||
if mounted:
|
||||
command(['umount', str(mount)])
|
||||
mounted = False
|
||||
if not found:
|
||||
report['errors'].append('Command directory unavailable: dirty/encrypted/unsupported filesystem or no flushed evidence')
|
||||
report['complete'] = found and not report['errors']
|
||||
except Exception as error:
|
||||
report['errors'].append(type(error).__name__ + ': ' + str(error)[:500])
|
||||
finally:
|
||||
if mounted:
|
||||
try:
|
||||
command(['umount', str(mount)])
|
||||
except Exception:
|
||||
report['errors'].append('Unmount incomplete; preserve stopped extractor')
|
||||
report['complete'] = False
|
||||
atomic_json(target / 'manifest.json', report)
|
||||
print(json.dumps({'command_id': cid, 'complete': report['complete']}))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Executable
+171
@@ -0,0 +1,171 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only live allow/deny probes for disposable owner bindings.
|
||||
Never executes guest code, changes a VM, creates a console or mutates PVE.
|
||||
Requires two existing disposable clone IDs, protected source ID and private
|
||||
storage identities. HTTP 500/404 is NOT a successful permission denial.
|
||||
Writes private evidence and a short-lived worker isolation proof only if every
|
||||
probe passes and administrator-exported storage paths prove nonalias isolation.
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import ssl
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def read_json(path):
|
||||
return json.loads(Path(path).read_text(encoding='utf-8'))
|
||||
|
||||
|
||||
def verified_context(cafile):
|
||||
# Python 3.13 enables the optional X509_STRICT profile, which rejects older
|
||||
# valid PVE root CAs without keyUsage. Match Go/curl chain validation while
|
||||
# retaining CERT_REQUIRED, hostname checks, certificate dates and trust roots.
|
||||
context = ssl.create_default_context(cafile=cafile or None)
|
||||
context.verify_flags &= ~ssl.VERIFY_X509_STRICT
|
||||
return context
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--config', type=Path, required=True)
|
||||
parser.add_argument('--owner', required=True)
|
||||
parser.add_argument('--foreign-owner', required=True)
|
||||
parser.add_argument('--own-disposable', type=int, required=True)
|
||||
parser.add_argument('--foreign-disposable', type=int, required=True)
|
||||
parser.add_argument('--source-vmid', type=int, required=True)
|
||||
parser.add_argument('--storage-inventory', type=Path, required=True,
|
||||
help='Private administrator pvesh GET /storage JSON export; verifies distinct paths/pools')
|
||||
parser.add_argument('--evidence', type=Path, required=True)
|
||||
parser.add_argument('--proof', type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
config = read_json(args.config)
|
||||
owner, foreign = config['owners'][args.owner], config['owners'][args.foreign_owner]
|
||||
if args.owner == args.foreign_owner or owner['pool'] == foreign['pool']:
|
||||
parser.error('two distinct owners and pools required')
|
||||
if args.own_disposable == args.foreign_disposable or min(args.own_disposable,args.foreign_disposable) < 100:
|
||||
parser.error('two distinct existing disposable IDs required')
|
||||
protected = {7000,7001,args.source_vmid}
|
||||
protected.update(v['vmid'] for v in config['sources'].values())
|
||||
if config.get('extractor'):
|
||||
protected.add(config['extractor']['source_vmid'])
|
||||
if args.own_disposable in protected or args.foreign_disposable in protected:
|
||||
parser.error('probe clone IDs must not be protected sources/control')
|
||||
endpoint = owner['endpoint'].rstrip('/')
|
||||
if urllib.parse.urlparse(endpoint).scheme != 'https' or endpoint != foreign['endpoint'].rstrip('/'):
|
||||
parser.error('both bindings must use same verified HTTPS endpoint')
|
||||
context = verified_context(owner.get('ca_file'))
|
||||
results = []
|
||||
|
||||
def probe(role, path, expected, binding=None):
|
||||
cred = read_json((binding or owner)[role + '_file'])
|
||||
request = urllib.request.Request(endpoint + '/api2/json' + path,
|
||||
headers={'Authorization': 'PVEAPIToken=' + cred['token_id'] + '=' + cred['secret']})
|
||||
body = None
|
||||
try:
|
||||
with urllib.request.urlopen(request, context=context, timeout=20) as response:
|
||||
status = response.status
|
||||
raw = response.read(1_048_577)
|
||||
if len(raw) <= 1_048_576:
|
||||
body = json.loads(raw).get('data')
|
||||
except urllib.error.HTTPError as error:
|
||||
status = error.code
|
||||
except Exception:
|
||||
status = 0
|
||||
passed = status == expected and (expected != 200 or body is not None)
|
||||
results.append({'role': role, 'token_id': cred['token_id'], 'path': path, 'status': status, 'expected': expected, 'passed': passed})
|
||||
return body if passed else None
|
||||
|
||||
node = owner['node']
|
||||
own = '/nodes/' + node + '/qemu/' + str(args.own_disposable)
|
||||
other = '/nodes/' + foreign['node'] + '/qemu/' + str(args.foreign_disposable)
|
||||
pool = probe('provisioner', '/pools/' + owner['pool'], 200)
|
||||
owned_config = probe('provisioner', own + '/config?current=1', 200)
|
||||
membership = pool is not None and any(m.get('vmid') == args.own_disposable and m.get('node') == node and m.get('type') == 'qemu' for m in pool.get('members', []))
|
||||
tags = set((owned_config or {}).get('tags', '').split(';'))
|
||||
results.append({'check':'own disposable membership and owner tag','passed':membership and ('otche-owner-'+args.owner) in tags})
|
||||
foreign_pool = probe('provisioner', '/pools/' + foreign['pool'], 200, foreign)
|
||||
foreign_config = probe('runtime', other + '/config?current=1', 200, foreign)
|
||||
foreign_member = foreign_pool is not None and any(m.get('vmid') == args.foreign_disposable and m.get('node') == foreign['node'] and m.get('type') == 'qemu' for m in foreign_pool.get('members', []))
|
||||
results.append({'check':'foreign disposable exists in its own pool with owner tag','passed':foreign_member and ('otche-owner-'+args.foreign_owner) in set((foreign_config or {}).get('tags','').split(';'))})
|
||||
# Values in this API are propagation flags, not grant switches: even zero
|
||||
# means the privilege exists on this path. Never ignore false-valued keys.
|
||||
narrow = {'runtime': {'VM.Audit','VM.GuestAgent.Unrestricted'},
|
||||
'recorder': {'VM.Audit','VM.Console'}}
|
||||
permission_paths = ['/', '/vms', '/vms/' + str(args.own_disposable),
|
||||
'/pool/' + owner['pool'], '/nodes/' + node,
|
||||
'/storage/' + owner['iso_storage'], '/access']
|
||||
for role, allowed in narrow.items():
|
||||
for permission_path in permission_paths:
|
||||
permissions = probe(role, '/access/permissions?' + urllib.parse.urlencode({'path':permission_path}), 200)
|
||||
valid = isinstance(permissions, dict)
|
||||
granted = set()
|
||||
if valid:
|
||||
for path, privileges in permissions.items():
|
||||
if path != permission_path or not isinstance(privileges, dict):
|
||||
valid = False
|
||||
break
|
||||
for privilege, propagate in privileges.items():
|
||||
if type(propagate) not in (bool,int) or propagate not in (False,True,0,1):
|
||||
valid = False
|
||||
granted.add(privilege)
|
||||
allowed_here = allowed if permission_path in ('/vms/'+str(args.own_disposable),'/pool/'+owner['pool']) else set()
|
||||
results.append({'role':role,'check':'effective privilege allowlist','acl_path':permission_path,
|
||||
'granted':sorted(granted),'passed':valid and granted == allowed_here})
|
||||
if config.get('extractor'):
|
||||
extractor_path = '/nodes/' + config['extractor']['node'] + '/qemu/' + str(config['extractor']['source_vmid'])
|
||||
probe(role, extractor_path + '/config?current=1', 403)
|
||||
probe(role, extractor_path + '/agent/get-osinfo', 403)
|
||||
for role in ('runtime','recorder'):
|
||||
probe(role, own + '/config?current=1', 200)
|
||||
probe(role, other + '/config?current=1', 403)
|
||||
probe(role, '/nodes/' + node + '/qemu/' + str(args.source_vmid) + '/config?current=1', 403)
|
||||
probe(role, '/nodes/' + node + '/lxc/7000/config', 403)
|
||||
probe('runtime', own + '/agent/get-osinfo', 200)
|
||||
probe('runtime', other + '/agent/get-osinfo', 403)
|
||||
probe('runtime', '/nodes/' + node + '/qemu/' + str(args.source_vmid) + '/agent/get-osinfo', 403)
|
||||
probe('recorder', own + '/agent/get-osinfo', 403)
|
||||
probe('uploader', '/nodes/' + node + '/storage/' + owner['iso_storage'] + '/content?content=iso', 200)
|
||||
probe('uploader', '/nodes/' + foreign['node'] + '/storage/' + foreign['iso_storage'] + '/content?content=iso', 200, foreign)
|
||||
probe('uploader', '/nodes/' + foreign['node'] + '/storage/' + foreign['iso_storage'] + '/content?content=iso', 403)
|
||||
probe('uploader', '/nodes/' + node + '/storage/local/content?content=iso', 403)
|
||||
inventory = read_json(args.storage_inventory)
|
||||
if isinstance(inventory, dict):
|
||||
inventory = inventory.get('data', [])
|
||||
by_id = {item['storage']: item for item in inventory}
|
||||
selected = [by_id.get(owner['iso_storage']),by_id.get(foreign['iso_storage'])]
|
||||
paths = [os.path.normpath(item.get('path','')) if item else '' for item in selected]
|
||||
private = all(item and item.get('type') == 'dir' and item.get('path','').startswith('/') for item in selected)
|
||||
private = private and paths[0] != paths[1] and all(path != '/var/lib/vz' for path in paths)
|
||||
private = private and not any(Path(left) in Path(right).parents for left, right in ((paths[0], paths[1]), (paths[1], paths[0])))
|
||||
for item in inventory:
|
||||
other_path = os.path.normpath(item.get('path',''))
|
||||
if item.get('storage') not in (owner['iso_storage'],foreign['iso_storage']) and item.get('path') and any(other_path == path or Path(other_path) in Path(path).parents or Path(path) in Path(other_path).parents for path in paths):
|
||||
private = False
|
||||
results.append({'check':'administrator storage inventory distinct private ISO paths without aliases','passed':private})
|
||||
passed = all(item['passed'] for item in results)
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
evidence = {'recorded_at':now.isoformat(),'owner_id':args.owner,'foreign_owner_id':args.foreign_owner,'passed':passed,'probes':results,
|
||||
'tls': {'certificate_verification': 'required', 'hostname_verification': True, 'optional_x509_strict_profile': False},
|
||||
'limits':'Read-only authorization and live guest-agent availability probes, not Windows execution qualification, a destructive deny test or online network proof. Storage inventory is trusted operator input.'}
|
||||
encoded = json.dumps(evidence,sort_keys=True,indent=2).encode()
|
||||
os.umask(0o077)
|
||||
args.evidence.parent.mkdir(parents=True,exist_ok=True)
|
||||
with args.evidence.open('xb') as f:
|
||||
f.write(encoded)
|
||||
if not passed:
|
||||
raise SystemExit('Isolation probes failed; private evidence written, no worker proof published')
|
||||
proof = {'owner_id':args.owner,'endpoint':owner['endpoint'],'node':node,'pool':owner['pool'],'iso_storage':owner['iso_storage'],'disk_storage':owner['disk_storage'],'passed':True,
|
||||
'expires_at':(now+datetime.timedelta(days=1)).isoformat(),'evidence_sha256':hashlib.sha256(encoded).hexdigest()}
|
||||
with args.proof.open('x',encoding='utf-8') as f:
|
||||
json.dump(proof,f,indent=2)
|
||||
print('Observed read-only isolation checks passed; private evidence and one-day proof created.')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"sources": {
|
||||
"win11-stopped-revision": {
|
||||
"node": "pve-node",
|
||||
"vmid": 7001,
|
||||
"owner_id": "11111111-1111-4111-8111-111111111111",
|
||||
"seal_path": "/var/lib/otche/source-seals/win11.json",
|
||||
"cd_slot": "ide2",
|
||||
"max_disk_bytes": 108447924224,
|
||||
"allowed_bridges": []
|
||||
}
|
||||
},
|
||||
"owners": {
|
||||
"11111111-1111-4111-8111-111111111111": {
|
||||
"endpoint": "https://pve.example.invalid:8006",
|
||||
"ca_file": "/run/otche/pve-ca.pem",
|
||||
"node": "pve-node",
|
||||
"pool": "otche-owner-one",
|
||||
"iso_storage": "otche-iso-one",
|
||||
"disk_storage": "otche-disks-one",
|
||||
"provisioner_file": "/run/otche/owner-one/provisioner.json",
|
||||
"runtime_file": "/run/otche/owner-one/runtime.json",
|
||||
"recorder_file": "/run/otche/owner-one/recorder.json",
|
||||
"uploader_file": "/run/otche/owner-one/uploader.json",
|
||||
"housekeeping_file": "/run/otche/owner-one/housekeeping.json",
|
||||
"isolation_proof_file": "/run/otche/owner-one/isolation-proof.json",
|
||||
"min_storage_free_bytes": 21474836480,
|
||||
"min_memory_free_bytes": 12884901888,
|
||||
"max_owned_disk_bytes": 343597383680,
|
||||
"max_owned_artifact_bytes": 8589934592,
|
||||
"online": null
|
||||
}
|
||||
},
|
||||
"xorriso": "/usr/bin/xorriso",
|
||||
"concurrent": 1,
|
||||
"min_artifact_free_bytes": 4294967296,
|
||||
"max_artifact_bytes": 67108864,
|
||||
"max_video_bytes": 2147483648,
|
||||
"task_timeout_seconds": 1200,
|
||||
"prepare_timeout_seconds": 180,
|
||||
"collect_timeout_seconds": 60,
|
||||
"watchdog_required": true,
|
||||
"qualification_owner": "11111111-1111-4111-8111-111111111111",
|
||||
"benign_fixture": "/run/otche/fixtures/otche-benign.exe",
|
||||
"benign_sha256": "",
|
||||
"extractor": null
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
#requires -Version 5.1
|
||||
[CmdletBinding()]
|
||||
param()
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference='Stop'
|
||||
Import-Module "$PSScriptRoot\Otche.Common.psm1" -Force
|
||||
$root='C:\ProgramData\Otche'
|
||||
$pointer=Read-OtcheJson "$root\control\pending-collector.json"
|
||||
$manifest=Read-OtcheJson ([string]$pointer.manifest_path)
|
||||
Confirm-OtcheManifest $manifest
|
||||
$directory="$root\results\$($manifest.command_id)"
|
||||
$receipt=Read-OtcheJson "$directory\receipt.json"
|
||||
$since=([DateTime]::Parse($receipt.accepted_at)).ToUniversalTime().AddMinutes(-2)
|
||||
$clock=[Diagnostics.Stopwatch]::StartNew()
|
||||
$errors=New-Object 'Collections.Generic.List[string]'
|
||||
$detections=New-Object 'Collections.Generic.List[object]'
|
||||
$seen=New-Object 'Collections.Generic.HashSet[string]'
|
||||
$events=New-Object 'Collections.Generic.List[object]'
|
||||
$before=$null
|
||||
try {$before=Get-OtcheDefender} catch {$errors.Add('Defender baseline: '+$_.Exception.Message)}
|
||||
$environment=$null
|
||||
try {$environment=Get-OtcheEnvironment} catch {$errors.Add('Environment: '+$_.Exception.Message)}
|
||||
$boot=Get-OtcheBootId
|
||||
$runtimeStatus=$null
|
||||
$candidatePaths=New-Object 'Collections.Generic.List[string]'
|
||||
$candidatePaths.Add("$root\samples\$($manifest.command_id)\$($manifest.filename)")
|
||||
foreach ($drive in [IO.DriveInfo]::GetDrives()) {
|
||||
if ($drive.DriveType -eq [IO.DriveType]::CDRom -and $drive.IsReady -and $drive.VolumeLabel -ceq $manifest.iso_label) {$candidatePaths.Add((Join-Path (Join-Path $drive.RootDirectory.FullName 'sample') $manifest.filename))}
|
||||
}
|
||||
function Add-CollectionError([string]$Text) {if ($errors.Count -lt 32 -and !$errors.Contains($Text)) {$errors.Add($Text)}}
|
||||
function Test-ResourceMatch([string]$Text) {
|
||||
$decoded=[System.Net.WebUtility]::HtmlDecode($Text)
|
||||
foreach ($candidate in $candidatePaths) {if ($decoded -match ([regex]::Escape($candidate)+'(?:$|[;<\s"''])')) {return $true}}
|
||||
return $false
|
||||
}
|
||||
function Get-Stage([DateTime]$Time) {
|
||||
$r=Read-OtcheJson "$directory\receipt.json"
|
||||
if ($null -ne $runtimeStatus -and $null -ne $runtimeStatus.report.execution.finished_at -and $Time.ToUniversalTime() -ge [DateTime]::Parse($runtimeStatus.report.execution.finished_at).ToUniversalTime()) {return 'collection'}
|
||||
if ($null -ne $r.started_at -and $Time.ToUniversalTime() -ge [DateTime]::Parse($r.started_at).ToUniversalTime()) {return 'execution'}
|
||||
if ($Time.ToUniversalTime() -lt [DateTime]::Parse($r.accepted_at).ToUniversalTime()) {return 'delivery'}
|
||||
return 'preparation'
|
||||
}
|
||||
do {
|
||||
if ([IO.File]::Exists("$directory\status.json")) {try {$runtimeStatus=Read-OtcheJson "$directory\status.json"} catch {Add-CollectionError ('Runner status: '+$_.Exception.Message)}}
|
||||
$after=$null
|
||||
try {$after=Get-OtcheDefender} catch {Add-CollectionError ('Defender current: '+$_.Exception.Message)}
|
||||
try {
|
||||
$raw=@(Get-MpThreatDetection -ErrorAction Stop | Sort-Object InitialDetectionTime -Descending | Select-Object -First 256)
|
||||
$threats=@(Get-MpThreat -ErrorAction Stop | Select-Object -First 256)
|
||||
if ($raw.Count -eq 256 -or $threats.Count -eq 256) {Add-CollectionError 'Defender query bound reached'}
|
||||
foreach ($d in $raw) {
|
||||
$time=$d.InitialDetectionTime
|
||||
if (!$time -or $time.ToUniversalTime() -lt $since) {continue}
|
||||
$resources=@($d.Resources | Select-Object -First 4 | ForEach-Object {$v=[string]$_; if ($v.Length -gt 1024) {Add-CollectionError 'Detection resource truncated';$v.Substring(0,1024)} else {$v}})
|
||||
if (@($d.Resources).Count -gt 4) {Add-CollectionError 'Detection resource bound reached'}
|
||||
if (!($resources | Where-Object {Test-ResourceMatch $_})) {continue}
|
||||
$key='threat:'+([string]$d.DetectionID)
|
||||
if (!$seen.Add($key)) {continue}
|
||||
if ($detections.Count -ge 64) {Add-CollectionError 'Detection output bound reached';continue}
|
||||
$threat=$threats | Where-Object {$_.ThreatID -eq $d.ThreatID} | Select-Object -First 1
|
||||
$name=$(if ($threat) {[string]$threat.ThreatName} else {'Threat '+$d.ThreatID})
|
||||
$detections.Add([ordered]@{name=$name; id=[string]$d.DetectionID; action=('cleaning_action={0};success={1};status={2}' -f $d.CleaningActionID,$d.ActionSuccess,$d.ThreatStatusID); resources=$resources; timestamp=$time.ToUniversalTime().ToString('o'); stage=(Get-Stage $time); source='defender'})
|
||||
}
|
||||
} catch {Add-CollectionError ('Defender detections: '+$_.Exception.Message)}
|
||||
foreach ($source in @(@('Microsoft-Windows-Windows Defender/Operational','defender'),@('Microsoft-Windows-AppLocker/EXE and DLL','policy'),@('Microsoft-Windows-AppLocker/MSI and Script','policy'),@('Microsoft-Windows-CodeIntegrity/Operational','policy'))) {
|
||||
try {
|
||||
$queryErrors=@()
|
||||
$batch=@(Get-WinEvent -FilterHashtable @{LogName=$source[0];StartTime=$since} -MaxEvents 200 -ErrorAction SilentlyContinue -ErrorVariable queryErrors)
|
||||
foreach ($queryError in $queryErrors) {if ($queryError.FullyQualifiedErrorId -notlike 'NoMatchingEventsFound*') {Add-CollectionError ('Event log '+$source[0]+': '+$queryError.Exception.Message)}}
|
||||
if ($batch.Count -eq 200) {Add-CollectionError ('Event log bound reached: '+$source[0])}
|
||||
foreach ($event in $batch) {
|
||||
$xml=$event.ToXml()
|
||||
if (!(Test-ResourceMatch $xml)) {continue}
|
||||
$matchedResources=@($candidatePaths | Where-Object {$xml -match [regex]::Escape($_)})
|
||||
$key=$source[0]+':'+$event.RecordId
|
||||
if (!$seen.Add($key)) {continue}
|
||||
if ($events.Count -ge 128) {Add-CollectionError 'Event output bound reached';continue}
|
||||
if ($xml.Length -gt 4096) {$xml=$xml.Substring(0,4096); Add-CollectionError 'Event XML truncated'}
|
||||
$events.Add([ordered]@{id=$event.Id; record_id=$event.RecordId; provider=$event.ProviderName; timestamp=$event.TimeCreated.ToUniversalTime().ToString('o'); xml=$xml})
|
||||
# Detection/action events only; ordinary operational events are not proof of blocking.
|
||||
if (($source[1] -eq 'defender' -and $event.Id -in @(1116,1117,1118,1119,1121)) -or ($source[1] -eq 'policy' -and $event.Id -in @(8004,8007,8029,3077))) {
|
||||
if ($detections.Count -lt 64) {$detections.Add([ordered]@{name=$event.ProviderName; id=[string]$event.Id; action=('event '+$event.Id); resources=$matchedResources; timestamp=$event.TimeCreated.ToUniversalTime().ToString('o'); stage=(Get-Stage $event.TimeCreated); source=$source[1]})} else {Add-CollectionError 'Detection output bound reached'}
|
||||
}
|
||||
}
|
||||
} catch {Add-CollectionError ('Event log '+$source[0]+': '+$_.Exception.Message)}
|
||||
}
|
||||
$drift=$null -ne $before -and $null -ne $after -and $before.fingerprint -ne $after.fingerprint
|
||||
if ($drift) {Add-CollectionError 'Defender baseline drift during observation'}
|
||||
Write-OtcheJson "$directory\defender-events.json" @($events.ToArray())
|
||||
Write-OtcheJson "$directory\telemetry.json" ([ordered]@{updated_at=(Get-OtcheUtc);boot_id=$boot;before=$before;after=$after;drift=$drift;detections=@($detections.ToArray());environment=$environment;errors=@($errors.ToArray())})
|
||||
if ([IO.File]::Exists("$directory\result.json")) {Export-OtcheGrub $manifest $directory;break}
|
||||
Start-Sleep -Seconds 3
|
||||
} while ($clock.Elapsed.TotalSeconds -lt ([int]$manifest.settings.duration_seconds+240))
|
||||
@@ -0,0 +1,147 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Prepares a Windows 10/11 source for disposable Otche clones, without creating or changing accounts.
|
||||
.DESCRIPTION
|
||||
Run elevated in Windows PowerShell 5.1 with an explicitly selected EXISTING local administrator:
|
||||
.\Install-OtcheSource.ps1 -Credential (Get-Credential "$env:COMPUTERNAME\LabRunner") -EnableAutoLogon
|
||||
The operator must create/approve the dedicated local account separately. Its existing password is
|
||||
validated with LogonUser; this installer NEVER resets passwords, changes memberships, disables UAC,
|
||||
changes Defender settings/exclusions, or changes any execution-policy scope. No disks are formatted.
|
||||
|
||||
Restricted blocks this installer AND every runner script, even signed scripts. Do not use -Bypass,
|
||||
EncodedCommand, or pasted scriptblocks to evade it: those do not provide a supported runnable source.
|
||||
An authorized administrator must first establish an approved script-execution policy independently
|
||||
(e.g. organizational AllSigned with a trusted code-signing chain). Sign every shipped .ps1 and .psm1
|
||||
with that trusted code-signing certificate, then run normally. RemoteSigned still requires signatures
|
||||
on downloaded/Internet-zoned scripts. This installer does not unblock files. ConstrainedLanguage,
|
||||
AppLocker/WDAC rules preventing trusted Add-Type, Task Scheduler, PowerShell, or signed runner scripts
|
||||
must be addressed by the organization's approved policy; no bypass is attempted.
|
||||
|
||||
Autologon is opt-in and uses the LSA DefaultPassword private secret, not a plaintext registry password.
|
||||
Administrators/SYSTEM can retrieve LSA secrets; protect source disks, snapshots, backups and console.
|
||||
Existing autologon/password configurations are refused instead of overwritten. Interactive Limited
|
||||
and Highest tasks use the selected account's existing split token. A standard-only account cannot
|
||||
satisfy admin runs. Administrator logon and a full desktop must be verified after a normal reboot.
|
||||
|
||||
No samples or qualification controls are generated on the source. Shut down the prepared ordinary
|
||||
VM through your operator workflow; clone CURRENT disks full=1 without converting it to a template.
|
||||
Qualification uses isolated disposable clones, never the master. See Source-Setup.txt beside this file.
|
||||
#>
|
||||
#requires -Version 5.1
|
||||
#requires -RunAsAdministrator
|
||||
[CmdletBinding()]
|
||||
param([Parameter(Mandatory=$true)][System.Management.Automation.PSCredential]$Credential,[switch]$EnableAutoLogon)
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference='Stop'
|
||||
if ($PSVersionTable.PSEdition -ne 'Desktop' -or $PSVersionTable.PSVersion.Major -ne 5) {throw 'Windows PowerShell 5.1 is required'}
|
||||
if ([Environment]::Is64BitOperatingSystem -and ![Environment]::Is64BitProcess) {throw 'Use the native 64-bit Windows PowerShell host'}
|
||||
if ($ExecutionContext.SessionState.LanguageMode -ne 'FullLanguage') {throw 'Approved FullLanguage policy is required for fixed trusted native helpers'}
|
||||
if ((Get-ExecutionPolicy) -eq 'Restricted') {throw 'Restricted blocks installer and runner files; obtain approved organizational script policy first'}
|
||||
$root='C:\ProgramData\Otche'
|
||||
if ([IO.Directory]::Exists($root)) {throw 'Otche source directory already exists; review existing setup rather than overwrite it'}
|
||||
foreach ($name in @('Otche-user','Otche-admin','Otche-Collector','Otche-DesktopReady')) {if (Get-ScheduledTask -TaskName $name -ErrorAction SilentlyContinue) {throw "Existing task $name must be reviewed, not overwritten"}}
|
||||
$user=$Credential.UserName
|
||||
if ($user -match '^([^\\]+)\\(.+)$') {
|
||||
$domain=$Matches[1];$user=$Matches[2]
|
||||
if ($domain -ne '.' -and $domain -ine $env:COMPUTERNAME) {throw 'Credential must name an existing local account, not a domain account'}
|
||||
} elseif ($user.Contains('@')) {throw 'Credential must name an existing local account'}
|
||||
$account=Get-LocalUser -Name $user -ErrorAction Stop
|
||||
if (!$account.Enabled) {throw 'Selected local account is disabled'}
|
||||
$admins=@(Get-LocalGroupMember -SID 'S-1-5-32-544' -ErrorAction Stop)
|
||||
if (!($admins | Where-Object {$_.SID.Value -eq $account.SID.Value})) {throw 'Existing local administrator split-token account required; memberships are never changed'}
|
||||
$uac=[Microsoft.Win32.Registry]::LocalMachine.OpenSubKey('SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System')
|
||||
try {if ($uac.GetValue('EnableLUA',0) -ne 1) {throw 'UAC must already be enabled; installer does not change it'}} finally {$uac.Dispose()}
|
||||
if ($account.SID.Value.EndsWith('-500')) {throw 'Use an operator-approved existing non-built-in split-token account, not RID500 Administrator'}
|
||||
$files=@('Otche.Common.psm1','Invoke-Otche.ps1','Invoke-OtcheDispatch.ps1','Start-OtcheCommand.ps1','Collect-Otche.ps1','Test-OtcheReady.ps1')
|
||||
foreach ($name in $files) {
|
||||
$path=Join-Path $PSScriptRoot $name
|
||||
if (![IO.File]::Exists($path)) {throw "Required runner file missing: $name"}
|
||||
$signature=Get-AuthenticodeSignature -LiteralPath $path
|
||||
if ((Get-ExecutionPolicy) -eq 'AllSigned' -and $signature.Status -ne 'Valid') {throw "AllSigned requires a valid trusted signature: $name"}
|
||||
}
|
||||
Add-Type -TypeDefinition @'
|
||||
using System;
|
||||
using System.Runtime.InteropServices;
|
||||
namespace OtcheInstall {
|
||||
public static class Secret {
|
||||
[StructLayout(LayoutKind.Sequential)] struct Unicode { public ushort Length; public ushort MaximumLength; public IntPtr Buffer; }
|
||||
[StructLayout(LayoutKind.Sequential)] struct Attributes { public uint Length; public IntPtr RootDirectory; public IntPtr ObjectName; public uint AttributesValue; public IntPtr SecurityDescriptor; public IntPtr SecurityQualityOfService; }
|
||||
[DllImport("advapi32.dll",CharSet=CharSet.Unicode,SetLastError=true)] public static extern bool LogonUser(string name,string domain,IntPtr password,int type,int provider,out IntPtr token);
|
||||
[DllImport("kernel32.dll")] public static extern bool CloseHandle(IntPtr token);
|
||||
[DllImport("advapi32.dll")] static extern uint LsaOpenPolicy(IntPtr system,ref Attributes attributes,uint access,out IntPtr handle);
|
||||
[DllImport("advapi32.dll")] static extern uint LsaStorePrivateData(IntPtr handle,ref Unicode key,ref Unicode value);
|
||||
[DllImport("advapi32.dll")] static extern uint LsaRetrievePrivateData(IntPtr handle,ref Unicode key,out IntPtr value);
|
||||
[DllImport("advapi32.dll")] static extern uint LsaFreeMemory(IntPtr value);
|
||||
[DllImport("advapi32.dll")] static extern uint LsaClose(IntPtr handle);
|
||||
[DllImport("advapi32.dll")] static extern uint LsaNtStatusToWinError(uint status);
|
||||
static void Check(uint status) {if(status!=0)throw new System.ComponentModel.Win32Exception((int)LsaNtStatusToWinError(status));}
|
||||
public static void StoreNew(IntPtr password,int chars) {
|
||||
var a=new Attributes();a.Length=(uint)Marshal.SizeOf(typeof(Attributes));IntPtr handle;
|
||||
Check(LsaOpenPolicy(IntPtr.Zero,ref a,0x24,out handle));
|
||||
IntPtr keyBuffer=Marshal.StringToHGlobalUni("DefaultPassword");
|
||||
var key=new Unicode{Buffer=keyBuffer,Length=30,MaximumLength=32};
|
||||
try {
|
||||
IntPtr existing;uint status=LsaRetrievePrivateData(handle,ref key,out existing);
|
||||
if(status==0){LsaFreeMemory(existing);throw new InvalidOperationException("An existing DefaultPassword LSA secret must not be overwritten");}
|
||||
if(LsaNtStatusToWinError(status)!=2)Check(status);
|
||||
var value=new Unicode{Buffer=password,Length=checked((ushort)(chars*2)),MaximumLength=checked((ushort)((chars+1)*2))};
|
||||
Check(LsaStorePrivateData(handle,ref key,ref value));
|
||||
}finally{Marshal.FreeHGlobal(keyBuffer);LsaClose(handle);}
|
||||
}
|
||||
}
|
||||
}
|
||||
'@
|
||||
$password=[Runtime.InteropServices.Marshal]::SecureStringToGlobalAllocUnicode($Credential.Password)
|
||||
$token=[IntPtr]::Zero
|
||||
try {
|
||||
if (![OtcheInstall.Secret]::LogonUser($user,$env:COMPUTERNAME,$password,2,0,[ref]$token)) {throw [ComponentModel.Win32Exception]::new([Runtime.InteropServices.Marshal]::GetLastWin32Error())}
|
||||
[void][OtcheInstall.Secret]::CloseHandle($token);$token=[IntPtr]::Zero
|
||||
$winlogon=$null
|
||||
if ($EnableAutoLogon) {
|
||||
$winlogon=[Microsoft.Win32.Registry]::LocalMachine.OpenSubKey('SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon',$true)
|
||||
if ($winlogon.GetValue('AutoAdminLogon','0') -eq '1' -or $null -ne $winlogon.GetValue('DefaultPassword',$null)) {$winlogon.Dispose();throw 'Existing autologon/plaintext password configuration refused'}
|
||||
try {[OtcheInstall.Secret]::StoreNew($password,$Credential.Password.Length)} catch {$winlogon.Dispose();throw}
|
||||
}
|
||||
[void][IO.Directory]::CreateDirectory($root)
|
||||
# Protected inheritance: only SYSTEM/admins modify trusted control/code; account only reads.
|
||||
$acl=[Security.AccessControl.DirectorySecurity]::new();$acl.SetAccessRuleProtection($true,$false)
|
||||
foreach ($sid in @('S-1-5-18','S-1-5-32-544')) {$acl.AddAccessRule([Security.AccessControl.FileSystemAccessRule]::new([Security.Principal.SecurityIdentifier]::new($sid),'FullControl','ContainerInherit,ObjectInherit','None','Allow'))}
|
||||
$acl.AddAccessRule([Security.AccessControl.FileSystemAccessRule]::new($account.SID,'ReadAndExecute','ContainerInherit,ObjectInherit','None','Allow'))
|
||||
[IO.Directory]::SetAccessControl($root,$acl)
|
||||
foreach ($dir in @('runner','control','results','samples')) {[void][IO.Directory]::CreateDirectory("$root\$dir")}
|
||||
foreach ($dir in @('results','samples')) {
|
||||
$rw=[IO.Directory]::GetAccessControl("$root\$dir")
|
||||
$rw.AddAccessRule([Security.AccessControl.FileSystemAccessRule]::new($account.SID,'Modify','ContainerInherit,ObjectInherit','None','Allow'))
|
||||
[IO.Directory]::SetAccessControl("$root\$dir",$rw)
|
||||
}
|
||||
foreach ($name in $files) {[IO.File]::Copy((Join-Path $PSScriptRoot $name),"$root\runner\$name",$false)}
|
||||
Import-Module "$root\runner\Otche.Common.psm1" -Force
|
||||
Write-OtcheJson "$root\source.json" @{account_sid=$account.SID.Value;account="$env:COMPUTERNAME\$user";installed_at=(Get-OtcheUtc);runner_version='1.0.0';autologon=[bool]$EnableAutoLogon} -CreateNew
|
||||
$exe="$env:WINDIR\System32\WindowsPowerShell\v1.0\powershell.exe"
|
||||
$settings=New-ScheduledTaskSettingsSet -MultipleInstances IgnoreNew -ExecutionTimeLimit ([TimeSpan]::FromMinutes(30)) -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries
|
||||
foreach ($level in @('user','admin')) {
|
||||
$principal=New-ScheduledTaskPrincipal -UserId "$env:COMPUTERNAME\$user" -LogonType Interactive -RunLevel $(if ($level -eq 'admin') {'Highest'} else {'Limited'})
|
||||
$action=New-ScheduledTaskAction -Execute $exe -Argument "-NoProfile -NonInteractive -File `"$root\runner\Invoke-OtcheDispatch.ps1`" -Privilege $level" -WorkingDirectory "$root\runner"
|
||||
Register-ScheduledTask -TaskName ('Otche-'+$level) -Action $action -Principal $principal -Settings $settings | Out-Null
|
||||
}
|
||||
$principal=New-ScheduledTaskPrincipal -UserId "$env:COMPUTERNAME\$user" -LogonType Interactive -RunLevel Limited
|
||||
$action=New-ScheduledTaskAction -Execute $exe -Argument "-NoProfile -NonInteractive -WindowStyle Hidden -File `"$root\runner\Test-OtcheReady.ps1`" -Privilege user -DesktopProbe" -WorkingDirectory "$root\runner"
|
||||
$desktopSettings=New-ScheduledTaskSettingsSet -MultipleInstances IgnoreNew -ExecutionTimeLimit ([TimeSpan]::FromSeconds(15)) -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries
|
||||
Register-ScheduledTask -TaskName 'Otche-DesktopReady' -Action $action -Principal $principal -Settings $desktopSettings | Out-Null
|
||||
$principal=New-ScheduledTaskPrincipal -UserId 'SYSTEM' -LogonType ServiceAccount -RunLevel Highest
|
||||
$action=New-ScheduledTaskAction -Execute $exe -Argument "-NoProfile -NonInteractive -File `"$root\runner\Collect-Otche.ps1`"" -WorkingDirectory "$root\runner"
|
||||
Register-ScheduledTask -TaskName 'Otche-Collector' -Action $action -Principal $principal -Settings $settings | Out-Null
|
||||
$baseline=Get-OtcheBaseline
|
||||
Write-OtcheJson "$root\installed-baseline.json" $baseline -CreateNew
|
||||
if ($EnableAutoLogon) {
|
||||
try {
|
||||
$winlogon.SetValue('DefaultUserName',$user,[Microsoft.Win32.RegistryValueKind]::String)
|
||||
$winlogon.SetValue('DefaultDomainName',$env:COMPUTERNAME,[Microsoft.Win32.RegistryValueKind]::String)
|
||||
$winlogon.SetValue('AutoAdminLogon','1',[Microsoft.Win32.RegistryValueKind]::String)
|
||||
} finally {$winlogon.Dispose()}
|
||||
}
|
||||
[ordered]@{installed=$true;qualified=$false;baseline_readable=$baseline.qualified;prerequisite='Reboot manually, verify active split-token desktop and QGA, shut down source, then qualify isolated disposable clones';errors=$baseline.errors} | ConvertTo-Json -Depth 5
|
||||
} finally {
|
||||
if ($token -ne [IntPtr]::Zero) {[void][OtcheInstall.Secret]::CloseHandle($token)}
|
||||
[Runtime.InteropServices.Marshal]::ZeroFreeGlobalAllocUnicode($password)
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
#requires -Version 5.1
|
||||
[CmdletBinding()]
|
||||
param([Parameter(Mandatory=$true)][string]$ManifestPath)
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference='Stop'
|
||||
Import-Module "$PSScriptRoot\Otche.Common.psm1" -Force
|
||||
$root='C:\ProgramData\Otche'
|
||||
$manifest=Read-OtcheJson $ManifestPath
|
||||
Confirm-OtcheManifest $manifest
|
||||
if ([IO.Path]::GetFullPath($ManifestPath) -ine "$root\control\$($manifest.command_id).json") {throw 'Invalid fixed manifest path'}
|
||||
$directory="$root\results\$($manifest.command_id)"
|
||||
$receipt=Read-OtcheJson "$directory\receipt.json"
|
||||
# This durable exclusive claim is never removed, even after failure or an unexpected reboot.
|
||||
Write-OtcheJson "$directory\dispatch.json" @{claimed_at=(Get-OtcheUtc);command_id=$manifest.command_id} -CreateNew
|
||||
$errors=New-Object 'Collections.Generic.List[string]'
|
||||
$execution=[ordered]@{exit_code=$null;error='';actual_duration_seconds=$null;started_at=$null;finished_at=$null;user='';session_id=$null;pid=$null;path='';arguments=@();handler='';privilege=[string]$manifest.settings.privilege}
|
||||
$report=[ordered]@{execution=$execution;defender=[ordered]@{before=$null;after=$null;drift=$false;detections=@()};environment=$null;collection_errors=@()}
|
||||
$state=[ordered]@{command_id=$manifest.command_id;attempt_id=$manifest.attempt_id;job_id=$manifest.job_id;phase='preparing';outcome='pending';findings='unknown';telemetry='pending';started_at=$null;deadline_at=$null;finished_at=$null;error='';report=$report;artifacts=@()}
|
||||
$process=$null; $clock=$null; $telemetry=$null; $lastTelemetry=$null
|
||||
$stdoutTask=$null; $stderrTask=$null
|
||||
function Update-Telemetry {
|
||||
try {
|
||||
$script:telemetry=Read-OtcheJson "$directory\telemetry.json"
|
||||
if ($script:telemetry.boot_id -ne $receipt.boot_id) {throw 'Telemetry boot changed'}
|
||||
$script:lastTelemetry=[DateTime]::Parse($script:telemetry.updated_at).ToUniversalTime()
|
||||
$report.defender.before=$script:telemetry.before; $report.defender.after=$script:telemetry.after
|
||||
$report.defender.drift=$script:telemetry.drift; $report.defender.detections=@($script:telemetry.detections)
|
||||
$report.environment=$script:telemetry.environment
|
||||
foreach ($errorText in $script:telemetry.errors) {if (!$errors.Contains($errorText) -and $errors.Count -lt 32) {$errors.Add($errorText)}}
|
||||
} catch {if ($errors.Count -lt 32 -and !$errors.Contains($_.Exception.Message)) {$errors.Add($_.Exception.Message)}}
|
||||
}
|
||||
function Save-Status {
|
||||
$report.collection_errors=@($errors.ToArray())
|
||||
Write-OtcheJson "$directory\status.json" $state
|
||||
}
|
||||
function Update-Findings {
|
||||
if (@($report.defender.detections | Where-Object {$_.source -eq 'defender'}).Count -gt 0) {$state.findings='detected'}
|
||||
elseif ($errors.Count -eq 0 -and $null -ne $report.defender.before -and $null -ne $report.defender.after) {$state.findings='not_observed'}
|
||||
else {$state.findings='unknown'}
|
||||
}
|
||||
function Collect-BoundedFile([string]$Source,[string]$Destination,[int]$Limit) {
|
||||
if (![IO.File]::Exists($Source)) {return}
|
||||
$input=[IO.File]::Open($Source,[IO.FileMode]::Open,[IO.FileAccess]::Read,[IO.FileShare]::ReadWrite)
|
||||
$output=[IO.File]::Open($Destination,[IO.FileMode]::Create,[IO.FileAccess]::Write,[IO.FileShare]::Read)
|
||||
try {
|
||||
$buffer=New-Object byte[] 16384; $remaining=$Limit
|
||||
while ($remaining -gt 0) {$n=$input.Read($buffer,0,[Math]::Min($buffer.Length,$remaining)); if ($n -eq 0) {break}; $output.Write($buffer,0,$n); $remaining-=$n}
|
||||
$output.Flush($true)
|
||||
if ($input.Length -gt $Limit -and !$errors.Contains('MSI log truncated at bounded artifact limit')) {$errors.Add('MSI log truncated at bounded artifact limit')}
|
||||
} finally {$input.Dispose();$output.Dispose()}
|
||||
}
|
||||
Save-Status
|
||||
try {
|
||||
$session=Get-OtcheSession $manifest.settings.privilege -Current
|
||||
$execution.user=$session.user; $execution.session_id=$session.session_id
|
||||
if ((Get-OtcheBootId) -ne $receipt.boot_id) {throw 'INTERRUPTED: Guest rebooted after command acceptance'}
|
||||
$readyClock=[Diagnostics.Stopwatch]::StartNew()
|
||||
while (![IO.File]::Exists("$directory\telemetry.json") -and $readyClock.Elapsed.TotalSeconds -lt 30) {Start-Sleep -Milliseconds 250}
|
||||
Update-Telemetry
|
||||
if ($null -eq $report.defender.before -or !$report.defender.before.active -or $null -eq $report.environment -or $errors.Count -gt 0) {throw 'Required baseline unavailable/incomplete; source is unqualified'}
|
||||
$volume=[IO.DriveInfo]::new('C:\')
|
||||
if ($volume.DriveFormat -ne 'NTFS') {throw 'Sample isolation directory must be NTFS'}
|
||||
$sampleDirectory="$root\samples\$($manifest.command_id)"
|
||||
[void][IO.Directory]::CreateDirectory($sampleDirectory)
|
||||
$path=Join-Path $sampleDirectory $manifest.filename
|
||||
$execution.path=$path
|
||||
$discs=@([IO.DriveInfo]::GetDrives() | Where-Object {$_.DriveType -eq [IO.DriveType]::CDRom -and $_.IsReady -and $_.VolumeLabel -ceq $manifest.iso_label})
|
||||
if ($discs.Count -ne 1) {throw 'DELIVERY: Exactly one matching sample ISO is required'}
|
||||
$iso=Read-OtcheJson (Join-Path $discs[0].RootDirectory.FullName 'job.json')
|
||||
foreach ($field in @('job_id','sha256','filename','iso_label')) {if ([string]$iso.$field -cne [string]$manifest.$field) {throw "DELIVERY: ISO manifest mismatch: $field"}}
|
||||
$original=Join-Path (Join-Path $discs[0].RootDirectory.FullName 'sample') $manifest.filename
|
||||
try {
|
||||
if (!(Test-Path -LiteralPath $original -PathType Leaf)) {throw 'ISO sample is missing'}
|
||||
if ((Get-OtcheHash $original) -cne $manifest.sha256) {throw 'ISO original SHA256 differs from immutable upload'}
|
||||
[IO.File]::Copy($original,$path,$false)
|
||||
if ((Get-OtcheHash $path) -cne $manifest.sha256) {throw 'NTFS sample SHA256 differs from immutable upload'}
|
||||
if ($manifest.settings.set_zoneid) {[IO.File]::WriteAllText($path+':Zone.Identifier',"[ZoneTransfer]`r`nZoneId=3`r`n",[Text.Encoding]::ASCII)}
|
||||
# When set_zoneid is false, no stream is removed or unblocked.
|
||||
} catch {
|
||||
$deliveryError=$_.Exception.Message
|
||||
# Wait for evidence propagation once; never retry copy or dispatch after quarantine.
|
||||
for ($i=0;$i -lt 4;$i++) {Start-Sleep -Seconds 3;Update-Telemetry;Save-Status}
|
||||
Update-Findings
|
||||
if ($state.findings -eq 'detected') {$state.outcome='blocked_before_execution'} else {$state.outcome='delivery_error'}
|
||||
throw ('DELIVERY: '+$deliveryError)
|
||||
}
|
||||
if ((Get-OtcheProperty $manifest 'qualification' '') -eq 'eicar') {
|
||||
$controlClock=[Diagnostics.Stopwatch]::StartNew()
|
||||
do {Start-Sleep -Seconds 3;Update-Telemetry;Update-Findings;Save-Status} while ($state.findings -ne 'detected' -and $controlClock.Elapsed.TotalSeconds -lt [int]$manifest.settings.duration_seconds)
|
||||
if ($state.findings -eq 'detected') {$state.outcome='blocked_before_execution'} else {$state.outcome='error';$state.error='EICAR delivery control was not observed; never executed'}
|
||||
} else {
|
||||
if (![IO.File]::Exists($path)) {
|
||||
Start-Sleep -Seconds 3;Update-Telemetry;Update-Findings
|
||||
if ($state.findings -eq 'detected') {$state.outcome='blocked_before_execution'} else {$state.outcome='delivery_error'}
|
||||
throw 'Sample disappeared before dispatch; no retry'
|
||||
}
|
||||
$launch=Get-OtcheLaunch $path $manifest.settings $directory
|
||||
$execution.handler=$launch.handler; $execution.arguments=@($launch.arguments)
|
||||
$start=[Diagnostics.ProcessStartInfo]::new()
|
||||
$start.FileName=$launch.file; $start.Arguments=$launch.command_line; $start.WorkingDirectory=$sampleDirectory
|
||||
$start.UseShellExecute=$false; $start.CreateNoWindow=$false
|
||||
foreach ($key in $launch.environment.Keys) {$start.EnvironmentVariables[$key]=$launch.environment[$key]}
|
||||
# Redirection only for console/script handlers; GUI shells remain visibly interactive.
|
||||
$capture=$launch.handler -in @('powershell-file','cscript','cmd')
|
||||
$start.RedirectStandardOutput=$capture; $start.RedirectStandardError=$capture
|
||||
$process=[Diagnostics.Process]::new(); $process.StartInfo=$start
|
||||
try {
|
||||
if (!$process.Start()) {throw 'Process.Start returned no process'}
|
||||
} catch {
|
||||
$dispatchError=$_.Exception.Message
|
||||
Start-Sleep -Seconds 3;Update-Telemetry;Update-Findings
|
||||
if ($state.findings -eq 'detected') {$state.outcome='blocked_before_execution'}
|
||||
elseif (@($report.defender.detections | Where-Object {$_.source -eq 'policy'}).Count -gt 0) {$state.outcome='policy_blocked'}
|
||||
throw ('DISPATCH: '+$dispatchError)
|
||||
}
|
||||
$clock=[Diagnostics.Stopwatch]::StartNew()
|
||||
$started=[DateTime]::UtcNow
|
||||
$execution.started_at=$started.ToString('o'); $execution.pid=$process.Id
|
||||
$state.started_at=$execution.started_at; $state.deadline_at=$started.AddSeconds([int]$manifest.settings.duration_seconds).ToString('o')
|
||||
$receipt.started_at=$state.started_at; $receipt.deadline_at=$state.deadline_at; $receipt.pid=$execution.pid; $receipt.state='running'
|
||||
Write-OtcheJson "$directory\receipt.json" $receipt
|
||||
$state.phase='running'; $state.outcome='executed'
|
||||
if ($capture) {
|
||||
Initialize-OtcheNative
|
||||
$stdoutTask=[Otche.Native]::Capture($process.StandardOutput,"$directory\stdout.log",1048576)
|
||||
$stderrTask=[Otche.Native]::Capture($process.StandardError,"$directory\stderr.log",1048576)
|
||||
}
|
||||
do {
|
||||
if ($process.HasExited -and $null -eq $execution.exit_code) {
|
||||
$execution.exit_code=$process.ExitCode
|
||||
if ($execution.exit_code -lt 0) {$execution.error=('Process exception/status 0x{0:X8}; not evidence of Defender or policy blocking' -f $execution.exit_code)}
|
||||
if ($launch.handler -eq 'msiexec' -and $execution.exit_code -in @(3010,1641)) {$execution.error=$(if ($execution.exit_code -eq 3010) {'MSI completed; reboot required (3010)'} else {'MSI initiated reboot (1641); worker must classify a boot change as interrupted'})}
|
||||
}
|
||||
Update-Telemetry
|
||||
if ($launch.handler -eq 'msiexec') {Collect-BoundedFile "$directory\msi.log" "$directory\msi-collected.log" 4194304}
|
||||
$execution.actual_duration_seconds=[Math]::Round($clock.Elapsed.TotalSeconds,3)
|
||||
Save-Status
|
||||
$remaining=[int]$manifest.settings.duration_seconds-$clock.Elapsed.TotalSeconds
|
||||
if ($remaining -gt 0) {Start-Sleep -Milliseconds ([int][Math]::Min(3000,[Math]::Ceiling($remaining*1000)))}
|
||||
} while ($clock.Elapsed.TotalSeconds -lt [int]$manifest.settings.duration_seconds)
|
||||
if ($process.HasExited -and $null -eq $execution.exit_code) {
|
||||
$execution.exit_code=$process.ExitCode
|
||||
if ($execution.exit_code -lt 0) {$execution.error=('Process exception/status 0x{0:X8}; not evidence of Defender or policy blocking' -f $execution.exit_code)}
|
||||
if ($launch.handler -eq 'msiexec' -and $execution.exit_code -in @(3010,1641)) {$execution.error=$(if ($execution.exit_code -eq 3010) {'MSI completed; reboot required (3010)'} else {'MSI initiated reboot (1641); worker must classify a boot change as interrupted'})}
|
||||
}
|
||||
$execution.actual_duration_seconds=[Math]::Round($clock.Elapsed.TotalSeconds,3)
|
||||
$execution.finished_at=Get-OtcheUtc
|
||||
if (@($report.defender.detections | Where-Object {$_.source -eq 'policy'}).Count -gt 0 -and $null -ne $execution.exit_code -and $execution.exit_code -ne 0) {$state.outcome='policy_blocked'}
|
||||
}
|
||||
} catch {
|
||||
$state.error=$_.Exception.Message
|
||||
$execution.error=$state.error
|
||||
if ($state.outcome -eq 'pending') {
|
||||
if ($state.error.StartsWith('INCOMPATIBLE:')) {$state.outcome='incompatible'}
|
||||
elseif ($state.error.StartsWith('INTERRUPTED:')) {$state.outcome='interrupted'}
|
||||
elseif ($state.error.StartsWith('DELIVERY:')) {$state.outcome='delivery_error'}
|
||||
else {$state.outcome='error'}
|
||||
} elseif ($state.outcome -eq 'executed') {$state.outcome='error'}
|
||||
} finally {
|
||||
$state.phase='collecting';Save-Status
|
||||
if (@(Get-OtcheProperty $manifest.settings 'grub_paths' @()).Count -gt 0) {
|
||||
# This runs on the already verified selected interactive token, not SYSTEM.
|
||||
$grubAfter=$(if ($null -ne $execution.started_at) {[DateTime]::Parse($execution.started_at)} else {[DateTime]::Parse($receipt.accepted_at)}).ToUniversalTime().AddSeconds([int]$manifest.settings.duration_seconds)
|
||||
while ([DateTime]::UtcNow -lt $grubAfter) {Start-Sleep -Milliseconds 500;Update-Telemetry;Save-Status}
|
||||
$report.grub=Get-OtcheGrubSnapshot $manifest $directory
|
||||
}
|
||||
$endCollection=[DateTime]::UtcNow
|
||||
$wait=[Diagnostics.Stopwatch]::StartNew()
|
||||
do {Start-Sleep -Milliseconds 500;Update-Telemetry} while (($null -eq $lastTelemetry -or $lastTelemetry -lt $endCollection) -and $wait.Elapsed.TotalSeconds -lt 12)
|
||||
if ($null -eq $lastTelemetry -or ([DateTime]::UtcNow-$lastTelemetry).TotalSeconds -gt 12) {$errors.Add('SYSTEM collector did not provide a fresh final snapshot')}
|
||||
foreach ($task in @($stdoutTask,$stderrTask)) {
|
||||
if ($null -eq $task) {continue}
|
||||
if ($task.IsFaulted) {$errors.Add('Process log capture failed: '+$task.Exception.GetBaseException().Message)}
|
||||
elseif ($task.IsCompleted -and $task.Result -gt 1048576) {$errors.Add('Process log truncated at 1 MiB')}
|
||||
}
|
||||
Update-Findings
|
||||
if ($null -eq $report.defender.before -and $null -eq $report.defender.after) {$state.telemetry='unavailable'} elseif ($errors.Count -gt 0) {$state.telemetry='partial'} else {$state.telemetry='complete'}
|
||||
if ($null -ne $clock -and $null -eq $execution.finished_at) {$execution.actual_duration_seconds=[Math]::Round($clock.Elapsed.TotalSeconds,3);$execution.finished_at=Get-OtcheUtc}
|
||||
$state.finished_at=Get-OtcheUtc; $state.phase='finished';$report.collection_errors=@($errors.ToArray())
|
||||
$artifacts=New-Object 'Collections.Generic.List[object]'
|
||||
foreach ($name in @('receipt.json','dispatch.json','telemetry.json','defender-events.json','stdout.log','stderr.log','msi-collected.log')) {
|
||||
if ([IO.File]::Exists("$directory\$name")) {$artifacts.Add(@{path="$directory\$name";filename=$name;kind=$(if ($name -eq 'msi-collected.log') {'msi_log'} elseif ($name.EndsWith('.log')) {'execution_log'} else {'telemetry'});content_type=$(if ($name.EndsWith('.json')) {'application/json'} else {'text/plain'})})}
|
||||
}
|
||||
$state.artifacts=$artifacts.ToArray()
|
||||
Save-Status
|
||||
Write-OtcheJson "$directory\result.json" $state
|
||||
if ($null -ne $process) {$process.Dispose()}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
#requires -Version 5.1
|
||||
[CmdletBinding()]
|
||||
param([Parameter(Mandatory=$true)][ValidateSet('user','admin')][string]$Privilege)
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference='Stop'
|
||||
Import-Module "$PSScriptRoot\Otche.Common.psm1" -Force
|
||||
$pointer=Read-OtcheJson "C:\ProgramData\Otche\control\pending-$Privilege.json"
|
||||
$manifest=Read-OtcheJson $pointer.manifest_path
|
||||
Confirm-OtcheManifest $manifest
|
||||
if ($manifest.settings.privilege -ne $Privilege) {throw 'Prepared task privilege mismatch'}
|
||||
& "$PSScriptRoot\Invoke-Otche.ps1" -ManifestPath $pointer.manifest_path
|
||||
@@ -0,0 +1,36 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Creates one explicit qualification control ONLY inside an already allocated disposable guest.
|
||||
.DESCRIPTION
|
||||
Never called by the source installer. Never run on a master or control machine. Worker/operator must
|
||||
place control\disposable-qualification.json only after allocating a disposable clone; it contains
|
||||
{purpose:"qualification",machine_guid:"<observed clone MachineGuid>",expires_at:"<UTC timestamp>"}.
|
||||
Both that short-lived marker and an explicit -DisposableVM acknowledgement are required. The worker
|
||||
may alternatively create these known bytes on its isolated fixture builder and deliver through ISO;
|
||||
real user uploads are never replaced by a fixture. EICAR is delivery-only and must never be executed.
|
||||
#>
|
||||
#requires -Version 5.1
|
||||
[CmdletBinding()]
|
||||
param([Parameter(Mandatory=$true)][ValidateSet('benign','eicar')][string]$Kind,[Parameter(Mandatory=$true)][string]$OutputDirectory,[Parameter(Mandatory=$true)][switch]$DisposableVM)
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference='Stop'
|
||||
if (!$DisposableVM) {throw 'Explicit disposable-VM acknowledgement required'}
|
||||
Import-Module "$PSScriptRoot\Otche.Common.psm1" -Force
|
||||
$marker=Read-OtcheJson 'C:\ProgramData\Otche\control\disposable-qualification.json'
|
||||
$key=[Microsoft.Win32.Registry]::LocalMachine.OpenSubKey('SOFTWARE\Microsoft\Cryptography')
|
||||
try {$machine=[string]$key.GetValue('MachineGuid')} finally {$key.Dispose()}
|
||||
if ($marker.purpose -ne 'qualification' -or $marker.machine_guid -ne $machine -or [DateTime]::Parse($marker.expires_at).ToUniversalTime() -le [DateTime]::UtcNow) {throw 'Missing, mismatched or expired disposable qualification marker'}
|
||||
$output=[IO.Path]::GetFullPath($OutputDirectory)
|
||||
if ($output.StartsWith('C:\ProgramData\Otche\',[StringComparison]::OrdinalIgnoreCase)) {throw 'Fixtures must not be written into installed source/control/results directories'}
|
||||
if ([IO.Directory]::Exists($output)) {throw 'Use a new fixture directory; existing files are never overwritten'}
|
||||
[void][IO.Directory]::CreateDirectory($output)
|
||||
if ($Kind -eq 'benign') {
|
||||
$path=Join-Path $output 'otche-benign.exe'
|
||||
Add-Type -TypeDefinition 'public static class OtcheBenignControl { public static int Main(string[] args) { System.Console.WriteLine("Otche benign qualification control"); return 0; } }' -OutputAssembly $path -OutputType ConsoleApplication -ErrorAction Stop
|
||||
} else {
|
||||
$path=Join-Path $output 'eicar.com'
|
||||
# Official inert antivirus test string. Defender may quarantine during this write; that is evidence.
|
||||
$text='X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*'
|
||||
[IO.File]::WriteAllText($path,$text,[Text.Encoding]::ASCII)
|
||||
}
|
||||
[ordered]@{kind=$Kind;path=$path;delivery_only=($Kind -eq 'eicar');sha256=$(if ([IO.File]::Exists($path)) {Get-OtcheHash $path} else {$null});present=[IO.File]::Exists($path)} | ConvertTo-Json -Compress
|
||||
@@ -0,0 +1,495 @@
|
||||
Set-StrictMode -Version 2.0
|
||||
$script:OtcheVersion = '1.0.0'
|
||||
$script:OtcheRoot = 'C:\ProgramData\Otche'
|
||||
|
||||
function Read-OtcheJson([string]$Path, [int]$MaxBytes = 1048576) {
|
||||
$stream = [IO.File]::Open($Path, [IO.FileMode]::Open, [IO.FileAccess]::Read, ([IO.FileShare]::ReadWrite -bor [IO.FileShare]::Delete))
|
||||
try {
|
||||
if ($stream.Length -gt $MaxBytes) { throw 'JSON exceeds bounded control limit' }
|
||||
$reader = New-Object IO.StreamReader($stream, [Text.Encoding]::UTF8, $true)
|
||||
try { return ($reader.ReadToEnd() | ConvertFrom-Json -ErrorAction Stop) } finally { $reader.Dispose() }
|
||||
} finally { $stream.Dispose() }
|
||||
}
|
||||
function Write-OtcheJson([string]$Path, $Value, [switch]$CreateNew) {
|
||||
$bytes = [Text.UTF8Encoding]::new($false).GetBytes((ConvertTo-Json -InputObject $Value -Depth 18 -Compress))
|
||||
if ($bytes.Length -gt 2097152) { throw 'JSON exceeds output limit' }
|
||||
if ($CreateNew) {
|
||||
$file = [IO.File]::Open($Path, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::Read)
|
||||
try { $file.Write($bytes, 0, $bytes.Length); $file.Flush($true) } finally { $file.Dispose() }
|
||||
return
|
||||
}
|
||||
$temp = $Path + '.' + [guid]::NewGuid().ToString('N') + '.tmp'
|
||||
try {
|
||||
$file = [IO.File]::Open($temp, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None)
|
||||
try { $file.Write($bytes, 0, $bytes.Length); $file.Flush($true) } finally { $file.Dispose() }
|
||||
if ([IO.File]::Exists($Path)) { [IO.File]::Replace($temp, $Path, [System.Management.Automation.Language.NullString]::Value) } else { [IO.File]::Move($temp, $Path) }
|
||||
} finally { if ([IO.File]::Exists($temp)) { [IO.File]::Delete($temp) } }
|
||||
}
|
||||
function Get-OtcheHash([string]$Path) {
|
||||
$file = [IO.File]::OpenRead($Path); $hash = [Security.Cryptography.SHA256]::Create()
|
||||
try { return ([BitConverter]::ToString($hash.ComputeHash($file))).Replace('-', '').ToLowerInvariant() }
|
||||
finally { $file.Dispose(); $hash.Dispose() }
|
||||
}
|
||||
function Get-OtcheTextHash([string]$Text) {
|
||||
$hash = [Security.Cryptography.SHA256]::Create()
|
||||
try { return ([BitConverter]::ToString($hash.ComputeHash([Text.Encoding]::UTF8.GetBytes($Text)))).Replace('-', '').ToLowerInvariant() }
|
||||
finally { $hash.Dispose() }
|
||||
}
|
||||
function Get-OtcheUtc { return [DateTime]::UtcNow.ToString('o') }
|
||||
function Get-OtcheProperty($Object, [string]$Name, $Default = $null) {
|
||||
if ($null -ne $Object -and $null -ne $Object.PSObject.Properties[$Name]) { return $Object.$Name }
|
||||
return $Default
|
||||
}
|
||||
function Confirm-OtcheManifest($Manifest) {
|
||||
foreach ($key in @('command_id','attempt_id','job_id')) {
|
||||
$v = [string](Get-OtcheProperty $Manifest $key)
|
||||
if ($v -notmatch '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$') { throw "Invalid $key" }
|
||||
}
|
||||
if ([string]$Manifest.iso_label -cnotmatch '^OT[0-9A-F]{14}$') { throw 'Invalid ISO label' }
|
||||
if ([string]$Manifest.sha256 -notmatch '^[0-9a-f]{64}$') { throw 'Invalid SHA256' }
|
||||
$name = [string]$Manifest.filename
|
||||
if (!$name -or $name.Length -gt 180 -or $name -ne [IO.Path]::GetFileName($name) -or $name.IndexOfAny([IO.Path]::GetInvalidFileNameChars()) -ge 0 -or $name -match '[\. ]$' -or $name -match '^(?i:CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])(?:\.|$)') { throw 'Unsafe execution filename' }
|
||||
$s = $Manifest.settings
|
||||
foreach ($rule in @(@('privilege','user','admin'),@('architecture','auto','x86','x64'),@('wsh_host','cscript','wscript'),@('msi_ui','full','quiet','passive'),@('internet','offline','online'),@('args_mode','none','custom'),@('filename','original','random'),@('dll_mode','regsvr32','rundll32'))) {
|
||||
if ([string](Get-OtcheProperty $s $rule[0]) -notin $rule[1..($rule.Length-1)]) { throw ('Invalid setting ' + $rule[0]) }
|
||||
}
|
||||
if ($s.duration_seconds -notin @(30,60,90,120,180,300,600,900,1200)) { throw 'Invalid duration' }
|
||||
if ($s.set_zoneid -isnot [bool]) { throw 'set_zoneid must be boolean' }
|
||||
if ($null -eq $s.args -or $s.args -isnot [array] -or $s.args.Count -gt 64) { throw 'args must be an explicit bounded array' }
|
||||
foreach ($arg in $s.args) { if ($arg -isnot [string] -or $arg.Length -gt 4096 -or $arg.Contains([string][char]0)) { throw 'Invalid argument' } }
|
||||
$grub=@(Get-OtcheProperty $s 'grub_paths' @())
|
||||
if ($grub.Count -gt 32 -or ($null -ne $s.PSObject.Properties['grub_paths'] -and $s.grub_paths -isnot [array])) {throw 'Invalid Grub path array'}
|
||||
if ($grub.Count -gt 0) {
|
||||
Initialize-OtcheGrub;$bytes=0
|
||||
foreach ($p in $grub) {
|
||||
if ($p -isnot [string]) {throw 'Grub paths must be strings'}
|
||||
$length=[Text.Encoding]::UTF8.GetByteCount($p);$bytes+=$length
|
||||
if ($length -gt 1024 -or $bytes -gt 8192) {throw 'Grub paths exceed byte bound'}
|
||||
[Otche.Grub]::Validate($p)
|
||||
}
|
||||
}
|
||||
$q = [string](Get-OtcheProperty $Manifest 'qualification' '')
|
||||
if ($q -notin @('','benign','eicar')) { throw 'Invalid qualification control' }
|
||||
}
|
||||
function ConvertTo-OtcheNativeArgument([AllowEmptyString()][string]$Value) {
|
||||
# CommandLineToArgvW / CRT convention, including trailing backslashes.
|
||||
$b = New-Object Text.StringBuilder
|
||||
[void]$b.Append('"'); $slashes = 0
|
||||
foreach ($c in $Value.ToCharArray()) {
|
||||
if ($c -eq '\') { $slashes++; continue }
|
||||
if ($c -eq '"') { [void]$b.Append(('\' * ($slashes * 2 + 1))); [void]$b.Append('"') }
|
||||
else { [void]$b.Append(('\' * $slashes)); [void]$b.Append($c) }
|
||||
$slashes = 0
|
||||
}
|
||||
[void]$b.Append(('\' * ($slashes * 2))); [void]$b.Append('"')
|
||||
return $b.ToString()
|
||||
}
|
||||
function Join-OtcheNativeArguments([string[]]$Values) { return (($Values | ForEach-Object { ConvertTo-OtcheNativeArgument $_ }) -join ' ') }
|
||||
function Initialize-OtcheNative {
|
||||
if ('Otche.Native' -as [type]) { return }
|
||||
Add-Type -TypeDefinition @'
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.Text;
|
||||
using System.Threading.Tasks;
|
||||
using System.Runtime.InteropServices;
|
||||
namespace Otche {
|
||||
public static class Native {
|
||||
[DllImport("kernel32.dll")] public static extern uint WTSGetActiveConsoleSessionId();
|
||||
[DllImport("wtsapi32.dll", CharSet=CharSet.Unicode, SetLastError=true)] static extern bool WTSQuerySessionInformation(IntPtr server, int session, int info, out IntPtr buffer, out int bytes);
|
||||
[DllImport("wtsapi32.dll")] static extern void WTSFreeMemory(IntPtr memory);
|
||||
[DllImport("advapi32.dll", SetLastError=true)] public static extern bool GetTokenInformation(IntPtr token, int type, out int value, int size, out int length);
|
||||
[DllImport("kernel32.dll")] public static extern bool CloseHandle(IntPtr handle);
|
||||
[DllImport("user32.dll")] public static extern IntPtr GetShellWindow();
|
||||
[DllImport("user32.dll",CharSet=CharSet.Unicode)] public static extern IntPtr FindWindow(string className,string title);
|
||||
[DllImport("user32.dll")] public static extern bool IsWindowVisible(IntPtr window);
|
||||
[DllImport("user32.dll")] public static extern uint GetWindowThreadProcessId(IntPtr window,out uint process);
|
||||
[DllImport("user32.dll",SetLastError=true)] static extern IntPtr OpenInputDesktop(uint flags,bool inherit,uint access);
|
||||
[DllImport("user32.dll")] static extern bool CloseDesktop(IntPtr desktop);
|
||||
[DllImport("user32.dll",CharSet=CharSet.Unicode,SetLastError=true)] static extern bool GetUserObjectInformation(IntPtr handle,int index,StringBuilder text,int length,out int needed);
|
||||
public delegate bool WindowCallback(IntPtr window,IntPtr state);
|
||||
[DllImport("user32.dll")] public static extern bool EnumWindows(WindowCallback callback,IntPtr state);
|
||||
public static string InputDesktop() {var h=OpenInputDesktop(0,false,1);if(h==IntPtr.Zero)throw new System.ComponentModel.Win32Exception();try{var b=new StringBuilder(256);int n;if(!GetUserObjectInformation(h,2,b,512,out n))throw new System.ComponentModel.Win32Exception();return b.ToString();}finally{CloseDesktop(h);}}
|
||||
public static string SessionValue(int session,int type) { IntPtr p; int n; if(!WTSQuerySessionInformation(IntPtr.Zero,session,type,out p,out n)) throw new System.ComponentModel.Win32Exception(); try{return Marshal.PtrToStringUni(p) ?? "";}finally{WTSFreeMemory(p);} }
|
||||
public static bool IsActive(int session) { IntPtr p; int n; if(!WTSQuerySessionInformation(IntPtr.Zero,session,8,out p,out n)) return false; try{return Marshal.ReadInt32(p)==0;}finally{WTSFreeMemory(p);} }
|
||||
public static Task<long> Capture(StreamReader reader,string path,int limit) {
|
||||
return Task.Run(async delegate { long total=0; using(var f=new FileStream(path,FileMode.CreateNew,FileAccess.Write,FileShare.Read)){var chars=new char[2048]; int written=0,n; while((n=await reader.ReadAsync(chars,0,chars.Length))!=0){var b=Encoding.UTF8.GetBytes(chars,0,n); total+=b.Length; int keep=Math.Min(b.Length,Math.Max(0,limit-written)); if(keep>0){await f.WriteAsync(b,0,keep); await f.FlushAsync(); written+=keep;}} f.Flush(true);} return total; });
|
||||
}
|
||||
}
|
||||
}
|
||||
'@ -ErrorAction Stop
|
||||
}
|
||||
function Get-OtcheSession([string]$Privilege, [switch]$Current) {
|
||||
Initialize-OtcheNative
|
||||
$config = Read-OtcheJson "$script:OtcheRoot\source.json"
|
||||
$sid = [int][Otche.Native]::WTSGetActiveConsoleSessionId()
|
||||
if ($sid -lt 1 -or ![Otche.Native]::IsActive($sid)) { throw 'No active interactive console session' }
|
||||
$account = [Otche.Native]::SessionValue($sid,7) + '\' + [Otche.Native]::SessionValue($sid,5)
|
||||
$accountSid = ([Security.Principal.NTAccount]::new($account)).Translate([Security.Principal.SecurityIdentifier]).Value
|
||||
if ($accountSid -ne $config.account_sid) { throw 'Active console account differs from prepared account' }
|
||||
if ($Current) {
|
||||
$identity = [Security.Principal.WindowsIdentity]::GetCurrent()
|
||||
$process = [Diagnostics.Process]::GetCurrentProcess()
|
||||
if ($identity.User.Value -ne $config.account_sid -or $process.SessionId -ne $sid -or ![Environment]::UserInteractive) { throw 'Runner is not using the prepared interactive token/session' }
|
||||
$elevated = 0; $size = 0
|
||||
if (![Otche.Native]::GetTokenInformation($identity.Token,20,[ref]$elevated,4,[ref]$size)) { throw 'Cannot inspect actual elevation token' }
|
||||
if (($Privilege -eq 'admin') -ne ($elevated -ne 0)) { throw 'Actual token privilege does not match request' }
|
||||
Assert-OtcheDesktop $sid
|
||||
}
|
||||
return [ordered]@{ session_id=$sid; user=$account; privilege=$Privilege }
|
||||
}
|
||||
function Assert-OtcheDesktop([int]$SessionId) {
|
||||
if ([Otche.Native]::InputDesktop() -cne 'Default') {throw 'Interactive input desktop is locked or secure'}
|
||||
$shell=[Otche.Native]::GetShellWindow();$taskbar=[Otche.Native]::FindWindow('Shell_TrayWnd',$null)
|
||||
if ($shell -eq [IntPtr]::Zero -or $taskbar -eq [IntPtr]::Zero -or ![Otche.Native]::IsWindowVisible($taskbar)) {throw 'Explorer desktop/taskbar is not ready; complete first-login setup'}
|
||||
foreach ($window in @($shell,$taskbar)) {
|
||||
[uint32]$owner=0;[void][Otche.Native]::GetWindowThreadProcessId($window,[ref]$owner)
|
||||
$process=Get-Process -Id $owner -ErrorAction Stop
|
||||
if ($process.ProcessName -ine 'explorer' -or $process.SessionId -ne $SessionId) {throw 'Desktop shell does not belong to the prepared Explorer session'}
|
||||
}
|
||||
$blocked=New-Object 'Collections.Generic.List[string]'
|
||||
$callback=[Otche.Native+WindowCallback]{param($window,$state)
|
||||
if ([Otche.Native]::IsWindowVisible($window)) {
|
||||
[uint32]$owner=0;[void][Otche.Native]::GetWindowThreadProcessId($window,[ref]$owner)
|
||||
$process=Get-Process -Id $owner -ErrorAction SilentlyContinue
|
||||
if ($process -and $process.ProcessName -in @('CloudExperienceHost','oobe','msoobe','LogonUI')) {$blocked.Add($process.ProcessName)}
|
||||
}
|
||||
return $true
|
||||
}
|
||||
[void][Otche.Native]::EnumWindows($callback,[IntPtr]::Zero)
|
||||
if ($blocked.Count) {throw ('Visible setup/sign-in host prevents desktop readiness: '+($blocked -join ', '))}
|
||||
}
|
||||
function Get-OtcheEnvironment {
|
||||
$key = [Microsoft.Win32.Registry]::LocalMachine.OpenSubKey('SOFTWARE\Microsoft\Windows NT\CurrentVersion')
|
||||
try { $build = '{0}.{1}.{2}' -f $key.GetValue('CurrentBuildNumber'),$key.GetValue('UBR'),$key.GetValue('BuildLabEx') } finally { $key.Dispose() }
|
||||
return [ordered]@{ os_build=$build; architecture=$(if ([Environment]::Is64BitOperatingSystem) {'x64'} else {'x86'}); powershell_version=$PSVersionTable.PSVersion.ToString(); execution_policy=[string](Get-ExecutionPolicy); runner_version=$script:OtcheVersion }
|
||||
}
|
||||
function Get-OtcheDefender {
|
||||
$status = Get-MpComputerStatus -ErrorAction Stop
|
||||
$prefs = Get-MpPreference -ErrorAction Stop
|
||||
$selected = [ordered]@{}
|
||||
foreach ($name in @('DisableRealtimeMonitoring','DisableBehaviorMonitoring','DisableIOAVProtection','DisableScriptScanning','DisableArchiveScanning','DisableBlockAtFirstSeen','MAPSReporting','SubmitSamplesConsent','PUAProtection','CloudBlockLevel','CloudExtendedTimeout','EnableControlledFolderAccess','AttackSurfaceReductionRules_Ids','AttackSurfaceReductionRules_Actions','ExclusionPath','ExclusionProcess','ExclusionExtension','ExclusionIpAddress')) {
|
||||
if ($null -eq $prefs.PSObject.Properties[$name]) { throw "Required Defender preference unreadable: $name" }
|
||||
$selected[$name] = $prefs.$name
|
||||
}
|
||||
$updated = $status.AntivirusSignatureLastUpdated
|
||||
if ($null -eq $updated) { throw 'Defender intelligence timestamp unavailable' }
|
||||
$state = [ordered]@{ active=([bool]$status.AMServiceEnabled -and [bool]$status.AntivirusEnabled -and [bool]$status.RealTimeProtectionEnabled -and !$prefs.DisableRealtimeMonitoring); platform=[string]$status.AMProductVersion; engine=[string]$status.AMEngineVersion; intelligence=[string]$status.AntivirusSignatureVersion; intelligence_updated_at=$updated.ToUniversalTime().ToString('o'); fingerprint=''; preferences=$selected }
|
||||
$state.fingerprint = Get-OtcheTextHash ($state | ConvertTo-Json -Depth 8 -Compress)
|
||||
return $state
|
||||
}
|
||||
function Get-OtcheBaseline {
|
||||
$errors = New-Object 'Collections.Generic.List[string]'; $defender=$null; $environment=$null
|
||||
try { $environment=Get-OtcheEnvironment } catch { $errors.Add($_.Exception.Message) }
|
||||
try { $defender=Get-OtcheDefender; if (!$defender.active) { $errors.Add('Defender is not active') } } catch { $errors.Add($_.Exception.Message) }
|
||||
try {
|
||||
$null=Get-MpThreatDetection -ErrorAction Stop
|
||||
$null=Get-MpThreat -ErrorAction Stop
|
||||
foreach ($channel in @('Microsoft-Windows-Windows Defender/Operational','Microsoft-Windows-AppLocker/EXE and DLL','Microsoft-Windows-AppLocker/MSI and Script','Microsoft-Windows-CodeIntegrity/Operational')) {$null=Get-WinEvent -ListLog $channel -ErrorAction Stop}
|
||||
} catch {$errors.Add('Required telemetry source unreadable: '+$_.Exception.Message)}
|
||||
$scripts=[ordered]@{}
|
||||
foreach ($name in @('Otche.Common.psm1','Invoke-Otche.ps1','Invoke-OtcheDispatch.ps1','Start-OtcheCommand.ps1','Collect-Otche.ps1','Test-OtcheReady.ps1')) {
|
||||
try {
|
||||
$path="$script:OtcheRoot\runner\$name"
|
||||
$signature=Get-AuthenticodeSignature -LiteralPath $path -ErrorAction Stop
|
||||
$scripts[$name]=[ordered]@{sha256=(Get-OtcheHash $path); signature=[string]$signature.Status; signer=$(if ($signature.SignerCertificate) {$signature.SignerCertificate.Thumbprint} else {''})}
|
||||
if ((Get-ExecutionPolicy) -eq 'AllSigned' -and $signature.Status -ne 'Valid') {$errors.Add("Required signature invalid: $name")}
|
||||
} catch { $errors.Add("Runner fingerprint $name unreadable: " + $_.Exception.Message) }
|
||||
}
|
||||
$policy=@(Get-ExecutionPolicy -List | ForEach-Object { [ordered]@{scope=[string]$_.Scope; policy=[string]$_.ExecutionPolicy} })
|
||||
if ((Get-ExecutionPolicy) -eq 'Restricted') { $errors.Add('Restricted blocks all script files, including signed runners') }
|
||||
$fingerprint=Get-OtcheTextHash ([ordered]@{environment=$environment; defender=$defender; scripts=$scripts; policies=$policy} | ConvertTo-Json -Depth 12 -Compress)
|
||||
return [ordered]@{fingerprint=$fingerprint; qualified=($errors.Count -eq 0); errors=@($errors.ToArray()); defender=$defender; environment=$environment; scripts=$scripts; policies=$policy}
|
||||
}
|
||||
function Get-OtcheBootId { return (Get-CimInstance Win32_OperatingSystem -ErrorAction Stop).LastBootUpTime.ToUniversalTime().ToString('o') }
|
||||
function Get-OtchePE([string]$Path) {
|
||||
$stream=[IO.File]::OpenRead($Path); $r=[IO.BinaryReader]::new($stream)
|
||||
try {
|
||||
if ($stream.Length -lt 64 -or $r.ReadUInt16() -ne 0x5a4d) { return $null }
|
||||
$stream.Position=60; $offset=$r.ReadUInt32()
|
||||
if ($offset -gt $stream.Length-24) { return $null }
|
||||
$stream.Position=$offset
|
||||
if ($r.ReadUInt32() -ne 0x4550) { return $null }
|
||||
$machine=$r.ReadUInt16(); $count=$r.ReadUInt16(); $stream.Position+=12; $optionalSize=$r.ReadUInt16(); $flags=$r.ReadUInt16()
|
||||
if ($count -gt 96 -or $optionalSize -lt 112 -or $offset+24+$optionalSize+$count*40 -gt $stream.Length) { throw 'Malformed PE header' }
|
||||
$optional=$stream.Position; $magic=$r.ReadUInt16()
|
||||
if ($magic -eq 0x10b) {$stream.Position=$optional+96} elseif ($magic -eq 0x20b) {$stream.Position=$optional+112} else {throw 'Unsupported PE optional header'}
|
||||
$exportRva=$r.ReadUInt32(); $exportSize=$r.ReadUInt32(); $sections=@()
|
||||
$stream.Position=$optional+$optionalSize
|
||||
for ($i=0;$i -lt $count;$i++) { $stream.Position+=8; $virtualSize=$r.ReadUInt32(); $rva=$r.ReadUInt32(); $rawSize=$r.ReadUInt32(); $raw=$r.ReadUInt32(); $stream.Position+=16; $sections+=@{rva=$rva; size=[Math]::Max($virtualSize,$rawSize); raw=$raw; raw_size=$rawSize} }
|
||||
$exports=New-Object 'Collections.Generic.List[string]'
|
||||
if ($exportRva -ne 0) {
|
||||
$location=Convert-OtcheRva $exportRva $sections $stream.Length; $stream.Position=$location+16
|
||||
$ordinalBase=$r.ReadUInt32(); $functionCount=$r.ReadUInt32(); $nameCount=$r.ReadUInt32(); $functions=$r.ReadUInt32(); $names=$r.ReadUInt32(); $ordinals=$r.ReadUInt32()
|
||||
if ($functionCount -gt 65536 -or $nameCount -gt 65536) {throw 'PE export table exceeds bound'}
|
||||
$functionOffset=Convert-OtcheRva $functions $sections $stream.Length
|
||||
for ($i=0;$i -lt $functionCount;$i++) { $stream.Position=$functionOffset+4*$i; if ($r.ReadUInt32() -ne 0) {$exports.Add('#'+($ordinalBase+$i))} }
|
||||
if ($nameCount -gt 0) {
|
||||
$namesOffset=Convert-OtcheRva $names $sections $stream.Length
|
||||
for ($i=0;$i -lt $nameCount;$i++) {
|
||||
$stream.Position=$namesOffset+4*$i; $nameRva=$r.ReadUInt32(); $stream.Position=Convert-OtcheRva $nameRva $sections $stream.Length
|
||||
$bytes=New-Object 'Collections.Generic.List[byte]'
|
||||
for ($j=0;$j -lt 256;$j++) {$v=$r.ReadByte(); if ($v -eq 0) {break}; $bytes.Add($v)}
|
||||
if ($j -eq 256) {throw 'PE export name exceeds bound'}
|
||||
$exports.Add([Text.Encoding]::ASCII.GetString($bytes.ToArray()))
|
||||
}
|
||||
}
|
||||
}
|
||||
return @{architecture=$(if ($machine -eq 0x14c) {'x86'} elseif ($machine -eq 0x8664) {'x64'} else {'unsupported'}); dll=(($flags -band 0x2000) -ne 0); exports=$exports.ToArray()}
|
||||
} finally {$r.Dispose(); $stream.Dispose()}
|
||||
}
|
||||
function Convert-OtcheRva([long]$Rva,$Sections,[long]$Length) {
|
||||
foreach ($s in $Sections) { if ($Rva -ge $s.rva -and $Rva-$s.rva -lt $s.raw_size) { $p=$s.raw+$Rva-$s.rva; if ($p -ge 0 -and $p -lt $Length) {return $p} } }
|
||||
throw 'PE RVA is outside file-backed sections'
|
||||
}
|
||||
function ConvertTo-OtcheBatchArgument([AllowEmptyString()][string]$Value, [switch]$Last) {
|
||||
# cmd's batch parameter parser is not CommandLineToArgvW. Preserve data with
|
||||
# one outer quote pair; escape only cmd operators exposed by inner quote pairs.
|
||||
$encoded=[Text.StringBuilder]::new(); [void]$encoded.Append('"'); $quoted=$true
|
||||
foreach ($c in $Value.ToCharArray()) {
|
||||
if ($c -eq '"') {$quoted=!$quoted;[void]$encoded.Append($c);continue}
|
||||
if ($c -eq [char]0 -or $c -eq "`r" -or $c -eq "`n") {throw 'INCOMPATIBLE: Batch parameters cannot contain NUL or line breaks'}
|
||||
if (!$quoted -and $c -in @(' ', "`t", ',', ';', '=')) {throw 'INCOMPATIBLE: Embedded quotes expose a batch argument separator; exact literal value is not representable'}
|
||||
if (!$quoted -and $c -in @('^','&','|','<','>','(',')')) {[void]$encoded.Append('^')}
|
||||
[void]$encoded.Append($c)
|
||||
}
|
||||
if (!$quoted -and !$Last) {throw 'INCOMPATIBLE: An unmatched literal quote would consume subsequent batch parameters'}
|
||||
[void]$encoded.Append('"');return $encoded.ToString()
|
||||
}
|
||||
function ConvertTo-OtcheMsiProperty([string]$Value) {
|
||||
$equal=$Value.IndexOf('=')
|
||||
return $Value.Substring(0,$equal)+'="'+$Value.Substring($equal+1).Replace('"','""')+'"'
|
||||
}
|
||||
function Get-OtcheLaunch([string]$Path,$Settings,[string]$Directory) {
|
||||
$ext=[IO.Path]::GetExtension($Path).ToLowerInvariant(); $arch=[string]$Settings.architecture
|
||||
$values=New-Object 'Collections.Generic.List[string]'
|
||||
if ($Settings.args_mode -eq 'custom') { foreach ($arg in $Settings.args) {$values.Add($arg.Replace('{sample.path}',$Path).Replace('{sample.name}',[IO.Path]::GetFileName($Path)))} }
|
||||
if ($ext -in @('.exe','.dll','.scr','.com')) {
|
||||
$pe=Get-OtchePE $Path
|
||||
if ($null -eq $pe) {throw 'INCOMPATIBLE: Non-PE/DOS16 executable is unsupported'}
|
||||
if ($pe.architecture -eq 'unsupported' -or ($pe.architecture -eq 'x64' -and ![Environment]::Is64BitOperatingSystem)) {throw 'INCOMPATIBLE: Unsupported PE architecture'}
|
||||
if ($arch -ne 'auto' -and $arch -ne $pe.architecture) {throw 'INCOMPATIBLE: Requested architecture disagrees with PE'}
|
||||
$arch=$pe.architecture
|
||||
}
|
||||
if ($arch -eq 'auto') {$arch=$(if ([Environment]::Is64BitOperatingSystem) {'x64'} else {'x86'})}
|
||||
if ($arch -eq 'x64' -and ![Environment]::Is64BitOperatingSystem) {throw 'INCOMPATIBLE: x64 handler unavailable'}
|
||||
$system=Join-Path $env:WINDIR 'System32'
|
||||
if ($arch -eq 'x86' -and [Environment]::Is64BitOperatingSystem) {$system=Join-Path $env:WINDIR 'SysWOW64'}
|
||||
$file=''; $arguments=@(); $handler=''; $raw=$null; $environment=@{}
|
||||
switch ($ext) {
|
||||
{$_ -in @('.exe','.scr','.com')} {if ($pe.dll) {throw 'INCOMPATIBLE: Executable has DLL characteristics'}; $file=$Path; $arguments=$values.ToArray(); $handler='direct'}
|
||||
'.dll' {
|
||||
if (!$pe.dll) {throw 'INCOMPATIBLE: File is not a PE DLL'}
|
||||
if ($Settings.dll_mode -eq 'regsvr32') {
|
||||
if ($pe.exports -cnotcontains 'DllRegisterServer') {throw 'INCOMPATIBLE: regsvr32 requires actual DllRegisterServer export'}
|
||||
if ($values.Count -ne 0) {throw 'INCOMPATIBLE: regsvr32 does not accept generic sample arguments'}
|
||||
$file=Join-Path $system 'regsvr32.exe'; $arguments=@($Path); $handler='regsvr32'
|
||||
} else {
|
||||
$export=[string]$Settings.dll_export
|
||||
if ($export -notmatch '^(?:[A-Za-z_?][A-Za-z0-9_?@$]*|#[0-9]+)$' -or $pe.exports -cnotcontains $export) {throw 'INCOMPATIBLE: Explicit DLL export is absent or invalid'}
|
||||
if ($Path.Contains(',')) {throw 'INCOMPATIBLE: rundll32 cannot safely address a comma in the DLL path'}
|
||||
$file=Join-Path $system 'rundll32.exe'; $arguments=@($Path+','+$export)+$values.ToArray(); $handler='rundll32'
|
||||
$raw='"'+$Path+'",'+$export
|
||||
if ($values.Count -gt 0) {$raw+=' '+(Join-OtcheNativeArguments $values.ToArray())}
|
||||
}
|
||||
}
|
||||
'.ps1' {$file=Join-Path $system 'WindowsPowerShell\v1.0\powershell.exe'; $arguments=@('-NoProfile','-File',$Path)+$values.ToArray(); $handler='powershell-file'}
|
||||
{$_ -in @('.vbs','.js')} {$file=Join-Path $system ($Settings.wsh_host+'.exe'); $arguments=@('//Nologo',$Path)+$values.ToArray(); $handler=$Settings.wsh_host}
|
||||
{$_ -in @('.bat','.cmd')} {
|
||||
$parts=@($Path)+$values.ToArray();$references=New-Object 'Collections.Generic.List[string]';$expandedLength=0
|
||||
for ($i=0;$i -lt $parts.Count;$i++) {
|
||||
$key='OTCHE_BATCH_ARG_'+$i
|
||||
$environment[$key]=ConvertTo-OtcheBatchArgument $parts[$i] -Last:($i -eq $parts.Count-1)
|
||||
$expandedLength+=$environment[$key].Length+1
|
||||
$references.Add('%'+$key+'%')
|
||||
}
|
||||
# Environment substitution is a single nonrecursive expansion: literal
|
||||
# percent expressions in values are not reparsed. Delayed expansion stays off.
|
||||
if ($expandedLength -gt 8000) {throw 'INCOMPATIBLE: Expanded batch command exceeds cmd.exe 8191-character limit'}
|
||||
$file=Join-Path $system 'cmd.exe'; $handler='cmd'; $arguments=$values.ToArray()
|
||||
$raw='/d /s /v:off /c "'+($references.ToArray() -join ' ')+'"'
|
||||
}
|
||||
'.msi' {
|
||||
$file=Join-Path $system 'msiexec.exe'; $handler='msiexec'
|
||||
$arguments=@('/i',$Path,'/L*V',(Join-Path $Directory 'msi.log'))
|
||||
$msiTokens=New-Object 'Collections.Generic.List[string]'
|
||||
$msiTokens.Add('/i');$msiTokens.Add('"'+$Path+'"');$msiTokens.Add('/L*V');$msiTokens.Add('"'+(Join-Path $Directory 'msi.log')+'"')
|
||||
$ui=$(if ($Settings.msi_ui -eq 'quiet') {'/qn'} elseif ($Settings.msi_ui -eq 'passive') {'/passive'} else {'/qf'})
|
||||
$arguments+=$ui;$msiTokens.Add($ui)
|
||||
for ($i=0;$i -lt $values.Count;$i++) {
|
||||
$arg=$values[$i]
|
||||
if ($arg -match '^[A-Za-z_][A-Za-z0-9_]*=') {
|
||||
$property=$arg.Substring(0,$arg.IndexOf('='))
|
||||
if ($property -in @('ACTION','UILEVEL')) {throw 'INCOMPATIBLE: MSI operation/UI is controlled by the selected sample/settings, not custom properties'}
|
||||
$arguments+=$arg;$msiTokens.Add((ConvertTo-OtcheMsiProperty $arg));continue
|
||||
}
|
||||
$option=$arg.ToLowerInvariant()
|
||||
if ($option -in @('/norestart','-norestart','/promptrestart','-promptrestart','/forcerestart','-forcerestart')) {
|
||||
if ($option -match 'promptrestart$' -and $Settings.msi_ui -eq 'quiet') {throw 'INCOMPATIBLE: msiexec does not support /promptrestart with quiet UI'}
|
||||
$arguments+=$arg;$msiTokens.Add($arg);continue
|
||||
}
|
||||
if ($option -in @('/m','-m','/n','-n')) {
|
||||
$i++;if ($i -ge $values.Count) {throw 'INCOMPATIBLE: MSI modifier requires an operand'}
|
||||
$operand=$values[$i]
|
||||
if ($option -in @('/m','-m') -and (!$operand -or $operand.Length -gt 8 -or $operand.IndexOfAny([IO.Path]::GetInvalidFileNameChars()) -ge 0 -or $operand -match '[. ]$')) {throw 'INCOMPATIBLE: MSI /m requires a valid SMS MIF basename of at most eight characters'}
|
||||
if ($option -in @('/n','-n') -and $operand -notmatch '^\{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\}$') {throw 'INCOMPATIBLE: MSI /n requires an instance ProductCode GUID'}
|
||||
$arguments+=@($arg,$operand);$msiTokens.Add($arg);$msiTokens.Add('"'+$operand+'"');continue
|
||||
}
|
||||
throw 'INCOMPATIBLE: MSI option is not an install modifier, or would replace the required sample, operation, log or selected UI'
|
||||
}
|
||||
$raw=$msiTokens.ToArray() -join ' '
|
||||
}
|
||||
default {throw 'INCOMPATIBLE: Unsupported sample extension'}
|
||||
}
|
||||
if (!$raw) {$raw=Join-OtcheNativeArguments $arguments}
|
||||
if ($raw.Length -gt 30000) {throw 'INCOMPATIBLE: Windows command line exceeds safe bound'}
|
||||
return @{file=$file; arguments=@($arguments); command_line=$raw; handler=$handler; architecture=$arch; environment=$environment}
|
||||
}
|
||||
function Initialize-OtcheGrub {
|
||||
if ('Otche.Grub' -as [type]) {return}
|
||||
Add-Type -TypeDefinition @'
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.Text;
|
||||
using System.Collections.Generic;
|
||||
using System.ComponentModel;
|
||||
using System.Runtime.InteropServices;
|
||||
using Microsoft.Win32.SafeHandles;
|
||||
namespace Otche {
|
||||
public static class Grub {
|
||||
public sealed class Capture {public long Size,OpenSize;public bool Changed;}
|
||||
[StructLayout(LayoutKind.Sequential)] struct Info {public uint Attributes; public System.Runtime.InteropServices.ComTypes.FILETIME Creation,Access,Write; public uint Volume,SizeHigh,SizeLow,Links,IndexHigh,IndexLow;}
|
||||
[DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] static extern SafeFileHandle CreateFile(string p,uint access,uint share,IntPtr security,uint disposition,uint flags,IntPtr template);
|
||||
[DllImport("kernel32.dll",SetLastError=true)] static extern bool GetFileInformationByHandle(SafeFileHandle h,out Info info);
|
||||
[DllImport("kernel32.dll")] static extern uint GetFileType(SafeFileHandle h);
|
||||
[DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] static extern uint GetFinalPathNameByHandle(SafeFileHandle h,StringBuilder path,uint length,uint flags);
|
||||
[DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] static extern bool GetVolumeInformationByHandleW(SafeFileHandle h,StringBuilder volume,uint volumeLength,out uint serial,out uint maxComponent,out uint flags,StringBuilder fs,uint fsLength);
|
||||
static void CheckLocalHandle(SafeFileHandle h,string expectedDevice,bool root) {
|
||||
var path=new StringBuilder(32768);uint count=GetFinalPathNameByHandle(h,path,(uint)path.Capacity,2);
|
||||
if(count==0 || count>=path.Capacity || !path.ToString().StartsWith(expectedDevice+"\\",StringComparison.OrdinalIgnoreCase))throw new ArgumentException("Opened handle is not on the verified local volume");
|
||||
if(root){uint serial,component,flags;var fs=new StringBuilder(64);if(!GetVolumeInformationByHandleW(h,null,0,out serial,out component,out flags,fs,(uint)fs.Capacity) || fs.ToString()!="NTFS")throw new ArgumentException("Opened volume is not NTFS");}
|
||||
}
|
||||
[DllImport("kernel32.dll",CharSet=CharSet.Unicode,SetLastError=true)] static extern uint QueryDosDevice(string name,StringBuilder target,int size);
|
||||
public static void Validate(string p) {
|
||||
if(p==null || p.Length<4 || p.Length>1024 || !System.Text.RegularExpressions.Regex.IsMatch(p,@"^[A-Za-z]:\\") || p.Substring(3).IndexOfAny("/:*?\"<>|".ToCharArray())>=0) throw new ArgumentException("Exact literal local absolute file path required; no devices, streams or wildcards");
|
||||
foreach(char c in p) if(c<32 || c==127)throw new ArgumentException("Control character in path");
|
||||
foreach(string part in p.Substring(3).Split('\\')) if(part.Length==0 || part=="." || part==".." || part.EndsWith(".") || part.EndsWith(" ") || System.Text.RegularExpressions.Regex.IsMatch(part,@"^(CON|PRN|AUX|NUL|COM[1-9\u00b9\u00b2\u00b3]|LPT[1-9\u00b9\u00b2\u00b3])(?:\.|$)",System.Text.RegularExpressions.RegexOptions.IgnoreCase)) throw new ArgumentException("Unsafe device, traversal or directory path");
|
||||
}
|
||||
// Pin every ancestor without FILE_SHARE_DELETE; no path component may be
|
||||
// replaced between validation and opening the file. OPEN_REPARSE_POINT never
|
||||
// follows the final component. Only an actual local fixed NTFS volume is used.
|
||||
public static Capture Copy(string source,string destination,long limit,DateTime deadline) {
|
||||
Validate(source);
|
||||
var drive=new DriveInfo(source.Substring(0,3));
|
||||
var target=new StringBuilder(1024);
|
||||
if(QueryDosDevice(source.Substring(0,2),target,target.Capacity)==0 || !System.Text.RegularExpressions.Regex.IsMatch(target.ToString(),@"^\\Device\\HarddiskVolume[0-9]+$") || drive.DriveType!=DriveType.Fixed || drive.DriveFormat!="NTFS") throw new ArgumentException("Only local fixed NTFS volumes are allowed");
|
||||
var handles=new List<SafeFileHandle>();
|
||||
try {
|
||||
Validate(destination);
|
||||
var destinationDrive=new DriveInfo(destination.Substring(0,3));var destinationTarget=new StringBuilder(1024);
|
||||
if(QueryDosDevice(destination.Substring(0,2),destinationTarget,destinationTarget.Capacity)==0 || !System.Text.RegularExpressions.Regex.IsMatch(destinationTarget.ToString(),@"^\\Device\\HarddiskVolume[0-9]+$") || destinationDrive.DriveType!=DriveType.Fixed || destinationDrive.DriveFormat!="NTFS")throw new ArgumentException("Export destination must be local NTFS");
|
||||
string parent=destination.Substring(0,3);string[] destinations=destination.Substring(3).Split('\\');
|
||||
for(int j=-1;j<destinations.Length-1;j++) {
|
||||
if(j>=0)parent=Path.Combine(parent,destinations[j]);
|
||||
var h=CreateFile(parent,0x80u,3u,IntPtr.Zero,3,0x00200000u|0x02000000u,IntPtr.Zero);
|
||||
if(h.IsInvalid){int error=Marshal.GetLastWin32Error();h.Dispose();throw new Win32Exception(error);}
|
||||
handles.Add(h);Info info;
|
||||
if(!GetFileInformationByHandle(h,out info) || (info.Attributes&0x400)!=0 || (info.Attributes&0x10)==0)throw new ArgumentException("Unsafe export directory/reparse point");
|
||||
CheckLocalHandle(h,destinationTarget.ToString(),j==-1);
|
||||
}
|
||||
string current=source.Substring(0,3); string[] parts=source.Substring(3).Split('\\');
|
||||
for(int i=-1;i<parts.Length;i++) {
|
||||
if(i>=0)current=Path.Combine(current,parts[i]); bool last=i==parts.Length-1;
|
||||
var h=CreateFile(current,last?0x80000000u:0x80u,3u,IntPtr.Zero,3,0x00200000u|0x02000000u,IntPtr.Zero);
|
||||
if(h.IsInvalid){int error=Marshal.GetLastWin32Error();h.Dispose();throw new Win32Exception(error);}
|
||||
handles.Add(h); Info info;
|
||||
if(GetFileType(h)!=1 || !GetFileInformationByHandle(h,out info))throw new IOException("Not an ordinary disk file");
|
||||
if((info.Attributes&0x400)!=0)throw new ArgumentException("Reparse points are not allowed");
|
||||
CheckLocalHandle(h,target.ToString(),i==-1);
|
||||
if(!last && (info.Attributes&0x10)==0)throw new ArgumentException("Non-directory ancestor");
|
||||
if(last) {
|
||||
if((info.Attributes&0x10)!=0)throw new ArgumentException("Directories are not supported");
|
||||
if(info.Links!=1)throw new ArgumentException("Hard-linked files are not supported");
|
||||
long length=((long)info.SizeHigh<<32)|info.SizeLow;
|
||||
if(length>limit)throw new OverflowException("File exceeds per-file or remaining total byte limit");
|
||||
// Capture at most the open length; concurrent writers are permitted, never chased.
|
||||
using(var input=new FileStream(h,FileAccess.Read,65536))
|
||||
using(var output=new FileStream(destination,FileMode.CreateNew,FileAccess.Write,FileShare.Read))
|
||||
using(var first=System.Security.Cryptography.SHA256.Create())
|
||||
using(var second=System.Security.Cryptography.SHA256.Create()) {
|
||||
var buffer=new byte[65536];long copied=0;int n;
|
||||
while(copied<length && (n=input.Read(buffer,0,(int)Math.Min(buffer.Length,length-copied)))!=0){if(DateTime.UtcNow>deadline)throw new TimeoutException("Grub collection deadline expired");copied+=n;output.Write(buffer,0,n);first.TransformBlock(buffer,0,n,buffer,0);}
|
||||
first.TransformFinalBlock(new byte[0],0,0);
|
||||
input.Position=0;long checkedBytes=0;
|
||||
while(checkedBytes<copied && (n=input.Read(buffer,0,(int)Math.Min(buffer.Length,copied-checkedBytes)))!=0){if(DateTime.UtcNow>deadline)throw new TimeoutException("Grub snapshot verification deadline expired");checkedBytes+=n;second.TransformBlock(buffer,0,n,buffer,0);}
|
||||
second.TransformFinalBlock(new byte[0],0,0);Info after=new Info();
|
||||
bool changed=copied!=length || input.Length!=length || checkedBytes!=copied || Convert.ToBase64String(first.Hash)!=Convert.ToBase64String(second.Hash) || !GetFileInformationByHandle(h,out after);
|
||||
if(!changed)changed=info.Write.dwHighDateTime!=after.Write.dwHighDateTime || info.Write.dwLowDateTime!=after.Write.dwLowDateTime;
|
||||
output.Flush(true);return new Capture{Size=copied,OpenSize=length,Changed=changed};
|
||||
}
|
||||
}
|
||||
}
|
||||
throw new IOException("File was not opened");
|
||||
} finally {foreach(var h in handles)h.Dispose();}
|
||||
}
|
||||
}
|
||||
}
|
||||
'@ -ErrorAction Stop
|
||||
}
|
||||
function Get-OtcheGrubSnapshot($Manifest,[string]$Directory) {
|
||||
Initialize-OtcheGrub
|
||||
$paths=@(Get-OtcheProperty $Manifest.settings 'grub_paths' @())
|
||||
$limits=$Manifest.grub_limits
|
||||
if ($paths.Count -gt 32 -or $limits.total_bytes -lt 0 -or $limits.total_bytes -gt 33554432 -or $limits.file_bytes -lt 0 -or $limits.file_bytes -gt 8388608 -or $limits.seconds -lt 1 -or $limits.seconds -gt 150) {throw 'Invalid Grub bounds'}
|
||||
$deadline=[DateTime]::UtcNow.AddSeconds([int]$limits.seconds)
|
||||
$files=New-Object 'Collections.Generic.List[object]';[long]$total=0
|
||||
for ($i=0;$i -lt $paths.Count;$i++) {
|
||||
$requested=[string]$paths[$i];$destination=Join-Path $Directory ('grub-{0:000}.bin' -f ($i+1))
|
||||
$entry=[ordered]@{requested_path=$requested;resolved_path=$null;member=$null;status='error';error='';size=$null;sha256=$null;snapshot=$null}
|
||||
try {
|
||||
if ([DateTime]::UtcNow -gt $deadline) {throw [TimeoutException]::new('Grub collection deadline expired')}
|
||||
$captured=[Otche.Grub]::Copy($requested,$destination,[Math]::Min([long]$limits.file_bytes,[long]$limits.total_bytes-$total),$deadline)
|
||||
$entry.sha256=Get-OtcheHash $destination;$entry.size=$captured.Size;$entry.resolved_path=$requested;$entry.status='collected';$total+=$captured.Size
|
||||
$entry.snapshot=@{open_size=$captured.OpenSize;changed=$captured.Changed;consistency='best_effort'}
|
||||
if ($captured.Changed) {$entry.status='changed';$entry.error='File changed during bounded open-length snapshot; archived bytes are not an atomic snapshot'}
|
||||
} catch {
|
||||
$exception=$_.Exception.GetBaseException();$entry.error=$exception.Message
|
||||
if ($entry.error.Length -gt 1024) {$entry.error=$entry.error.Substring(0,1024)}
|
||||
if ($exception -is [ArgumentException]) {$entry.status='invalid_path'}
|
||||
elseif ($exception -is [OverflowException]) {$entry.status='oversize'}
|
||||
elseif ($exception -is [ComponentModel.Win32Exception]) {
|
||||
if ($exception.NativeErrorCode -in @(2,3)) {$entry.status='missing'}
|
||||
elseif ($exception.NativeErrorCode -eq 5) {$entry.status='access_denied'}
|
||||
elseif ($exception.NativeErrorCode -in @(32,33)) {$entry.status='changed'}
|
||||
} elseif ($exception -is [IO.IOException]) {$entry.status='changed'}
|
||||
if ([IO.File]::Exists($destination)) {[IO.File]::Delete($destination)}
|
||||
}
|
||||
$files.Add($entry)
|
||||
}
|
||||
$collected=@($files | Where-Object {$null -ne $_.size}).Count
|
||||
$complete=@($files | Where-Object {$_.status -eq 'collected'}).Count -eq $paths.Count
|
||||
return [ordered]@{status=$(if ($complete) {'complete'} elseif ($collected -eq 0) {'empty'} else {'partial'});requested=$paths.Count;collected=$collected;files=@($files.ToArray())}
|
||||
}
|
||||
function Export-OtcheGrub($Manifest,[string]$Directory) {
|
||||
Initialize-OtcheGrub
|
||||
$paths=@(Get-OtcheProperty $Manifest.settings 'grub_paths' @())
|
||||
if ($paths.Count -eq 0) {return}
|
||||
$result=Read-OtcheJson "$Directory\result.json"
|
||||
$prefix="$script:OtcheRoot\control\grub-$($Manifest.command_id)"
|
||||
try {
|
||||
if ($result.report.grub.files.Count -ne $paths.Count) {throw 'Grub export mapping is incomplete'}
|
||||
$total=0L;$deadline=[DateTime]::UtcNow.AddSeconds([int]$Manifest.grub_limits.seconds)
|
||||
for ($i=0;$i -lt $paths.Count;$i++) {
|
||||
$entry=$result.report.grub.files[$i]
|
||||
if ($entry.requested_path -cne $paths[$i]) {throw 'Grub export mapping mismatch'}
|
||||
if ($null -eq $entry.size) {continue}
|
||||
$source=Join-Path $Directory ('grub-{0:000}.bin' -f ($i+1))
|
||||
$destination=('{0}-{1:000}.bin' -f $prefix,($i+1))
|
||||
$captured=[Otche.Grub]::Copy($source,$destination,[Math]::Min([long]$Manifest.grub_limits.file_bytes,[long]$Manifest.grub_limits.total_bytes-$total),$deadline)
|
||||
if ($captured.Changed -or $captured.Size -ne $entry.size -or (Get-OtcheHash $destination) -cne $entry.sha256) {throw 'Grub staged file changed before export'}
|
||||
$total+=$captured.Size
|
||||
}
|
||||
Write-OtcheJson "$prefix.json" @{command_id=$Manifest.command_id;ready=$true;error=''} -CreateNew
|
||||
} catch {Write-OtcheJson "$prefix.json" @{command_id=$Manifest.command_id;ready=$false;error=$_.Exception.GetBaseException().Message} -CreateNew}
|
||||
}
|
||||
Export-ModuleMember -Function *-Otche*
|
||||
@@ -0,0 +1,185 @@
|
||||
OTCHE WINDOWS SOURCE SETUP / WORKER CONTRACT
|
||||
|
||||
Prerequisites and trust
|
||||
- Windows 10/11 with Windows PowerShell Desktop 5.1, NTFS C:, QEMU Guest Agent,
|
||||
working display driver, Task Scheduler, active Defender, and an existing dedicated
|
||||
local non-RID500 administrator account with its existing password. Operator selects
|
||||
it explicitly with Get-Credential. Installer neither creates nor modifies accounts.
|
||||
- UAC must already be enabled. Limited and Highest tasks use the same account's
|
||||
split interactive token. The user task is never elevated; admin is never silently
|
||||
downgraded. Actual account SID, console session and TokenElevation are checked.
|
||||
- Restricted blocks ALL script files, including signed installer and signed runner.
|
||||
First obtain an independently approved organizational script-execution policy.
|
||||
Recommended: AllSigned, with every supplied .ps1 and .psm1 signed by an approved
|
||||
code-signing certificate trusted by LocalMachine and the selected account. Verify
|
||||
signatures with Get-AuthenticodeSignature. Sign only after reviewing final files.
|
||||
RemoteSigned requires valid signatures on downloaded/Internet-zoned scripts.
|
||||
No policy switch, Unblock-File, EncodedCommand, paste-to-evade-policy, or Bypass is
|
||||
supported. Interactive pasting does not solve Restricted blocking scheduled files.
|
||||
AppLocker/WDAC/ConstrainedLanguage must independently permit the reviewed runner,
|
||||
native Add-Type helpers and selected execution hosts. Protection stays enabled.
|
||||
- Run elevated: .\Install-OtcheSource.ps1 -Credential (Get-Credential
|
||||
"$env:COMPUTERNAME\LabRunner") -EnableAutoLogon
|
||||
This command is one line. Omitting EnableAutoLogon leaves login to the operator.
|
||||
The password is validated, never reset; opt-in autologon stores only an LSA private
|
||||
secret. It refuses existing autologon/DefaultPassword configuration rather than
|
||||
overwriting it. LSA secrets remain recoverable by SYSTEM/admins: protect master,
|
||||
clones, exports and backups. Installation failures require operator inspection;
|
||||
installer does not destructively rollback accounts, registry or existing evidence.
|
||||
- No disk is initialized or formatted and no evidence disk is assumed. Artifacts
|
||||
use existing NTFS C:. The full source remains an ordinary stopped VM template=0.
|
||||
Operator reboots and verifies a real active console desktop before stopping it.
|
||||
Complete first-login privacy/setup through the real UI. An active WTS session alone
|
||||
is insufficient: the input desktop must be unlocked Default, with the expected
|
||||
account's real Explorer shell and visible taskbar, and no visible setup/sign-in host.
|
||||
Worker clones CURRENT disks with full=1, never a snapshot name. Never alter master.
|
||||
For a source without CD, provision an empty CD on the stopped CLONE, not the source.
|
||||
- These files do not establish network isolation. Worker must remove clone NICs
|
||||
BEFORE media delivery in offline mode and prove fail-closed isolation for online.
|
||||
Sample delivery follows recorder writable-sink + full-frame readiness only.
|
||||
|
||||
Worker commands (native System32 WindowsPowerShell\v1.0\powershell.exe):
|
||||
-NoProfile -NonInteractive -File C:\ProgramData\Otche\runner\Test-OtcheReady.ps1 -Privilege user
|
||||
-NoProfile -NonInteractive -File C:\ProgramData\Otche\runner\Start-OtcheCommand.ps1 -ManifestPath C:\ProgramData\Otche\control\<command UUID>.json
|
||||
Invoke-Otche.ps1 -ManifestPath is the installed interactive runner interface. QGA
|
||||
never invokes the sample or submits a generic shell command. The fixed scheduler
|
||||
starts prepared InteractiveToken tasks (Limited/Highest), plus a separate SYSTEM
|
||||
Defender collector. All commands and control JSON must be short/bounded; use ISO
|
||||
for samples and bounded QGA file-open/read offsets for artifacts, not repeated
|
||||
file-write overwrites. No credential, password or PVE token belongs in a manifest.
|
||||
Readiness uses a fixed Otche-DesktopReady InteractiveToken/Limited task running the
|
||||
signed Test-OtcheReady.ps1 -Privilege user -DesktopProbe with no command/input API.
|
||||
Each request has a new nonce and 12-second deadline; bounded response must match the
|
||||
nonce, boot ID, configured account and active console session. The task is limited
|
||||
to 15 seconds and runs without opening a console window. UserOOBEBroker residency
|
||||
and arbitrary WWAHost applications are not treated as unfinished setup.
|
||||
The normal readiness gate runs before media/dispatch; the actual runner repeats
|
||||
desktop/token checks once before launch. During observation, -BootOnly returns only
|
||||
the boot ID without launching a desktop probe. Post-run -BaselineOnly collects the
|
||||
full source/Defender baseline without reinterpreting sample-owned GUI as setup.
|
||||
|
||||
Manifest (JSON data, max 1 MiB): command_id, attempt_id, job_id (canonical UUIDs),
|
||||
iso_label OT<first 14 uppercase hexadecimal job UUID characters> (16 ASCII characters, identical ISO9660/Joliet labels), sha256 (lowercase),
|
||||
filename (execution_filename selected ONCE by worker per Job), settings from API.
|
||||
Settings require architecture auto|x86|x64, wsh_host cscript|wscript, msi_ui
|
||||
full|quiet|passive, plus existing fields. args is a JSON string array; only
|
||||
{sample.path}/{sample.name} are expanded. ISO root job.json contains job_id,
|
||||
sha256,filename,iso_label; payload is sample/<filename>, unchanged original bytes.
|
||||
Optional root qualification='benign'|'eicar'. EICAR is delivery-only, never started.
|
||||
Fixture generation is a separate explicit disposable-only script, never installer.
|
||||
Its short-lived machine-bound qualification marker is placed only on a clone.
|
||||
|
||||
Results: C:\ProgramData\Otche\results\<command UUID>\
|
||||
- receipt.json: command_id,attempt_id,job_id,state accepted|running,accepted_at,
|
||||
started_at,deadline_at,session_id,user,privilege,pid,boot_id.
|
||||
- status.json refreshed about every 3 seconds, result.json atomically terminal:
|
||||
command_id,attempt_id,job_id,phase,outcome,findings,telemetry,started_at,deadline_at,
|
||||
finished_at,error,report,artifacts[{path,filename,kind,content_type}]. report matches
|
||||
docs/API.md. Artifacts are bounded JSON/log files; preserve them if collection fails.
|
||||
- receipt CreateNew precedes scheduling; dispatch.json CreateNew precedes runner
|
||||
activity. One clone accepts exactly one command. Existing receipt => duplicate,
|
||||
NEVER rerun after failure/crash. accepted receipt has null started_at/deadline_at;
|
||||
these become real timestamps only after selected sample host Process.Start succeeds.
|
||||
Preparation failure/quarantine never invents startup/duration/PID. Worker enforces
|
||||
a preparation deadline and then external runtime from started_at/deadline_at.
|
||||
- Full observation uses Stopwatch and continues when the parent exits; descendants
|
||||
are not guessed from installed applications. Worker stops clone after collection.
|
||||
Unexpected reboot/disconnect is externally interrupted, never resumed/reset.
|
||||
- Test-OtcheReady returns ready,session_id,user,privilege,boot_id,baseline
|
||||
{fingerprint,qualified,errors},defender,environment,error. qualified here means
|
||||
readable active prerequisites, NOT a published qualified revision. Worker compares
|
||||
observed fingerprint with externally qualified revision; drift or unreadable
|
||||
required telemetry is unqualified. Baseline fingerprints include full OS build,
|
||||
runner hashes/signatures, PS policy and Defender versions/preferences. Intelligence
|
||||
updates can change fingerprints; requalify rather than silently accept drift.
|
||||
|
||||
Dispatch and evidence semantics
|
||||
- EXE/SCR/PE COM direct; DOS16/non-PE COM explicitly incompatible. PE x86/x64 chooses
|
||||
matching hosts; unsupported machines/architecture mismatch fail before dispatch.
|
||||
- DLL regsvr32 requires actual DllRegisterServer export. rundll32 requires explicitly
|
||||
requested actual named/ordinal export, never guessed. Existence does not establish
|
||||
a compatible ABI; operator must choose a rundll32-compatible export. Commas in DLL
|
||||
paths are rejected. Native CRT quoting is separate from rundll32's module syntax.
|
||||
- PS1 uses -File and no execution-policy bypass. WSH uses requested cscript/wscript.
|
||||
CMD/BAT uses fixed cmd /d /s /v:off /c with per-process environment transport and
|
||||
one nonrecursive percent substitution, not concatenation of user text into shell
|
||||
source. Delayed expansion remains OFF. Reports retain resolved original argument
|
||||
values, never internal environment-variable aliases.
|
||||
Actual CMD and BAT fixtures preserve spaces, Unicode, a&b, (parentheses), literal%,
|
||||
literal %PATH%, !bang!, caret^value, comma/semicolon/equal, empty strings, trailing
|
||||
backslashes, and balanced literal quotes such as say"hello". A terminal value a"b
|
||||
is also supported. These are measured cases, not a claim about every batch script's
|
||||
own argument processing (a script can itself reinterpret values through CALL/etc.).
|
||||
Native batch parameters have no general CRT-style quote escape. An unmatched quote
|
||||
in a NONFINAL value consumes following arguments: requesting [a"b, tail] yields
|
||||
first value a"b" "tail and no second value in the direct native control. The runner
|
||||
rejects that case. Inner quotes exposing spaces/tabs/comma/semicolon/equal also
|
||||
split native parameters: say "two words" is not one exact literal parameter.
|
||||
These specific boundaries, NUL/CR/LF, and the bounded cmd line limit fail explicitly;
|
||||
there is no blanket percent/metacharacter/quote rejection. A batch script needing
|
||||
otherwise unrepresentable text needs its own explicit data-file interface, not
|
||||
silently changed argument bytes. No generic shell API is added.
|
||||
- MSI keeps /i <selected sample>, required /L*V <command MSI log>, and selected UI.
|
||||
Explicit custom install modifiers /norestart, /promptrestart, /forcerestart,
|
||||
/m <SMS MIF basename <=8 characters>, /n <instance ProductCode GUID> are accepted,
|
||||
alongside PROPERTY=value entries. /promptrestart with quiet UI is natively invalid.
|
||||
Operation/package replacements (/x, another /i, /a, /j, /p, /y, /z, etc.), custom
|
||||
log/UI switches and ACTION/UILEVEL properties cannot replace controlled invariants.
|
||||
Property values use Windows Installer syntax, not CRT backslash escaping:
|
||||
COMPANY=Acme "Widgets" becomes COMPANY="Acme ""Widgets"""; empty values and trailing
|
||||
backslashes remain literal. /m requires the environment's real SMS ISMIF32.DLL;
|
||||
no successful status-file generation is claimed from command construction alone.
|
||||
No implicit installed-application launch. Reboots are suppressed ONLY if explicitly
|
||||
requested by the submitted arguments; actual reboot is externally interrupted.
|
||||
Exit 3010 means reboot required; 1641 means reboot initiated, not Defender blocking.
|
||||
Microsoft command syntax references:
|
||||
https://learn.microsoft.com/en-us/windows/win32/msi/command-line-options
|
||||
https://learn.microsoft.com/en-us/windows/win32/msi/standard-installer-command-line-options
|
||||
- SHA256 checks original ISO and isolated NTFS copy. ZoneId=3 is written only when on;
|
||||
off never removes a stream or unblocks. Missing file without correlated detection
|
||||
remains unknown/delivery_error. There is no retry after early quarantine.
|
||||
- SYSTEM collector records Defender state/preferences before and after, correlated
|
||||
detections and Defender/AppLocker/CodeIntegrity events by time and sample path.
|
||||
Ordinary operational/audit events are not declared blocks. No SmartScreen result
|
||||
is invented from a dialog/exit. Crash/status exit differs from policy evidence.
|
||||
- Detection lists cap 64 (4 resources x 1 KiB), event lists 128, queries 200/channel,
|
||||
Defender queries 256, event XML 4 KiB,
|
||||
control JSON 1 MiB, output JSON 2 MiB, stdout/stderr 1 MiB, exported MSI log 4 MiB.
|
||||
Raw MSI logging can grow during observation; disposable disk must have capacity.
|
||||
Truncated/unreadable evidence is partial, not clean. Sample processes share their
|
||||
runner token and admin samples can alter guest evidence; external video and worker
|
||||
timestamps remain independent. Guest artifacts are observations, not tamper-proof.
|
||||
|
||||
Verification boundary
|
||||
Local AST/harmless quoting/native helper smoke checks do not qualify a guest image.
|
||||
Only real disposable-clone readiness, recorded benign controls and EICAR delivery,
|
||||
with identical before/after fingerprint and collected evidence, qualify a revision.
|
||||
The reproducible harmless regression is Test-OtcheArguments.ps1 (run normally under
|
||||
the existing approved policy). It compiles a temporary benign environment reader,
|
||||
executes both .cmd/.bat fixtures, prints the native unmatched-quote counterexample,
|
||||
checks literal resolved argument values and MSI construction, and deletes its files.
|
||||
Run local tests only under an approved existing policy; they do not authorize source or protection changes.
|
||||
|
||||
Grub optional file snapshots
|
||||
- settings.grub_paths is an immutable bounded literal array: max32 paths, 1024 UTF8
|
||||
bytes/path, 8192 total. Exact local absolute DOS paths only; no environment/sample
|
||||
expansion. Percent/braces are ordinary filename characters. No directories/globs,
|
||||
UNC/devices/ADS/traversal, hard links or reparse points. Open handles must resolve
|
||||
to the verified actual local fixed NTFS volume, not merely a checked drive letter.
|
||||
- Invoke-Otche captures after the observation deadline on its existing selected
|
||||
interactive token. It shares reads/writes (not delete), pins ancestor handles,
|
||||
captures at most open length and compares a second bounded read plus size/write
|
||||
metadata. Concurrent readable logs work; detected changes retain explicit
|
||||
best_effort snapshots, never assert atomic consistency. Unknown size/hash are null;
|
||||
zero bytes is a real captured empty file. Per-file failures do not change Defender.
|
||||
- grub_limits is controller-only manifest data: per-file <=8MiB, total <=32MiB,
|
||||
further restricted by max_artifact_bytes/2 and collection deadline. No credentials.
|
||||
Numbered results/grub-NNN.bin are transient selected-user snapshots. SYSTEM only
|
||||
safe-opens those fixed files, verifies size/hash, copies into protected
|
||||
control/grub-<command>-NNN.bin and publishes a bounded export-ready receipt.
|
||||
Requested file paths are never read by SYSTEM or on the host. Raw staged copies
|
||||
are not individual artifacts; controller streams one ZIP with safe numbered
|
||||
members and a per-file manifest, commits it before allowed clone cleanup.
|
||||
- Test-OtcheGrub.ps1 is a harmless local helper regression (not source qualification):
|
||||
actual live writer/read sharing, identical basenames, empty/missing/denied files,
|
||||
literal percent/braces, byte limits and device/network/traversal/reparse rejection.
|
||||
@@ -0,0 +1,33 @@
|
||||
#requires -Version 5.1
|
||||
[CmdletBinding()]
|
||||
param([Parameter(Mandatory=$true)][string]$ManifestPath)
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference='Stop'
|
||||
Import-Module "$PSScriptRoot\Otche.Common.psm1" -Force
|
||||
$root='C:\ProgramData\Otche'
|
||||
try {
|
||||
$manifest=Read-OtcheJson $ManifestPath
|
||||
Confirm-OtcheManifest $manifest
|
||||
$expected="$root\control\$($manifest.command_id).json"
|
||||
if ([IO.Path]::GetFullPath($ManifestPath) -ine $expected) {throw 'Manifest path is not the fixed command path'}
|
||||
$identity=[Security.Principal.WindowsIdentity]::GetCurrent()
|
||||
if ($identity.User.Value -ne 'S-1-5-18' -and !([Security.Principal.WindowsPrincipal]::new($identity)).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) {throw 'Scheduler requires SYSTEM or elevated operator'}
|
||||
$directory="$root\results\$($manifest.command_id)"
|
||||
[void][IO.Directory]::CreateDirectory($directory)
|
||||
if ([IO.File]::Exists("$directory\receipt.json")) {
|
||||
[ordered]@{command_id=$manifest.command_id;state='duplicate';started_at=$null} | ConvertTo-Json -Compress
|
||||
exit 0
|
||||
}
|
||||
$session=Get-OtcheSession $manifest.settings.privilege
|
||||
# One disposable clone accepts one command for its lifetime. A crash never makes a replay safe.
|
||||
Write-OtcheJson "$root\control\allocation.json" @{command_id=$manifest.command_id} -CreateNew
|
||||
Write-OtcheJson "$directory\receipt.json" ([ordered]@{command_id=$manifest.command_id;attempt_id=$manifest.attempt_id;job_id=$manifest.job_id;state='accepted';accepted_at=(Get-OtcheUtc);started_at=$null;deadline_at=$null;session_id=$session.session_id;user=$session.user;privilege=$manifest.settings.privilege;pid=$null;boot_id=(Get-OtcheBootId)}) -CreateNew
|
||||
Write-OtcheJson "$root\control\pending-collector.json" @{manifest_path=$expected}
|
||||
Write-OtcheJson "$root\control\pending-$($manifest.settings.privilege).json" @{manifest_path=$expected}
|
||||
Start-ScheduledTask -TaskName 'Otche-Collector' -ErrorAction Stop
|
||||
Start-ScheduledTask -TaskName ('Otche-'+$manifest.settings.privilege) -ErrorAction Stop
|
||||
[ordered]@{command_id=$manifest.command_id;state='accepted';started_at=$null} | ConvertTo-Json -Compress
|
||||
} catch {
|
||||
[ordered]@{error=$_.Exception.Message} | ConvertTo-Json -Compress
|
||||
exit 1
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Harmless, reproducible Windows PowerShell 5.1 handler-boundary regression.
|
||||
.DESCRIPTION
|
||||
Creates a temporary benign console helper and CMD/BAT fixtures, verifies actual
|
||||
batch argument values and the native unmatched-quote limitation, then deletes
|
||||
all owned temporary files. MSI checks construct command lines only: no MSI is
|
||||
installed. No account, policy, registry, Defender or source configuration changes.
|
||||
Run under the existing approved script policy; never add an execution-policy switch.
|
||||
#>
|
||||
#requires -Version 5.1
|
||||
[CmdletBinding()]
|
||||
param()
|
||||
Set-StrictMode -Version 2
|
||||
$ErrorActionPreference='Stop'
|
||||
Import-Module "$PSScriptRoot\Otche.Common.psm1" -Force
|
||||
$root=Join-Path ([IO.Path]::GetTempPath()) ('otche-argv-'+[guid]::NewGuid().ToString('N'))
|
||||
[void][IO.Directory]::CreateDirectory($root)
|
||||
function Invoke-Fixture($Launch) {
|
||||
$process=[Diagnostics.Process]::new()
|
||||
$process.StartInfo=[Diagnostics.ProcessStartInfo]::new($Launch.file,$Launch.command_line)
|
||||
$process.StartInfo.UseShellExecute=$false
|
||||
$process.StartInfo.RedirectStandardOutput=$true
|
||||
$process.StartInfo.RedirectStandardError=$true
|
||||
foreach ($key in $Launch.environment.Keys) {$process.StartInfo.EnvironmentVariables[$key]=$Launch.environment[$key]}
|
||||
try {
|
||||
[void]$process.Start()
|
||||
$stdout=$process.StandardOutput.ReadToEnd();$stderr=$process.StandardError.ReadToEnd()
|
||||
if (!$process.WaitForExit(10000)) {$process.Kill();throw 'Harmless CMD fixture timed out'}
|
||||
if ($process.ExitCode -ne 0 -or $stderr) {throw ('Harmless CMD fixture failed: '+$stderr)}
|
||||
return @($stdout.TrimEnd("`r","`n") -split "`r?`n" | ForEach-Object {[Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($_))})
|
||||
} finally {$process.Dispose()}
|
||||
}
|
||||
try {
|
||||
$exe=Join-Path $root 'environment-reader.exe'
|
||||
Add-Type -TypeDefinition 'using System; public static class EchoEnv { public static int Main() { for (int i=0;i<12;i++) Console.WriteLine(Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(Environment.GetEnvironmentVariable("CAP"+i) ?? ""))); return 0; } }' -OutputAssembly $exe -OutputType ConsoleApplication
|
||||
$body="@echo off`r`nsetlocal DisableDelayedExpansion`r`n"
|
||||
for ($i=0;$i -lt 12;$i++) {$body+='set "CAP'+$i+'=%~1"'+"`r`nshift`r`n"}
|
||||
$body+='"'+$exe+'"'+"`r`n"
|
||||
# Unicode value assembled without requiring an encoding assumption for this script file.
|
||||
$unicode=([string][char]0x041f)+[char]0x0440+[char]0x0438+[char]0x0432+[char]0x0435+[char]0x0442+' '+[char]0x4e16+[char]0x754c
|
||||
$values=@('space value',$unicode,'a&b','(parentheses)','literal%','%PATH%','!OTCHE_UNSET!','say"hello"','caret^value','','comma,semi;equal=','trailing\')
|
||||
$settings=[pscustomobject]@{architecture='auto';args_mode='custom';args=$values;dll_mode='regsvr32';dll_export='';wsh_host='cscript';msi_ui='full'}
|
||||
foreach ($extension in @('cmd','bat')) {
|
||||
$batch=Join-Path $root ('spaces & percent% '+$unicode+' fixture.'+$extension)
|
||||
[IO.File]::WriteAllText($batch,$body,[Text.Encoding]::ASCII)
|
||||
$launch=Get-OtcheLaunch $batch $settings $root
|
||||
$actual=@(Invoke-Fixture $launch)
|
||||
if ((ConvertTo-Json -InputObject $actual -Compress) -cne (ConvertTo-Json -InputObject $values -Compress)) {throw ('CMD/BAT argument values changed: '+(ConvertTo-Json -InputObject $actual -Compress))}
|
||||
if ((ConvertTo-Json -InputObject $launch.arguments -Compress) -cne (ConvertTo-Json -InputObject $values -Compress)) {throw 'Report arguments exposed transport aliases rather than resolved input values'}
|
||||
}
|
||||
$settings.args=@('prefix','a"b')
|
||||
$terminalQuote=@(Invoke-Fixture (Get-OtcheLaunch $batch $settings $root))
|
||||
if ($terminalQuote.Count -ne 2 -or $terminalQuote[0] -cne 'prefix' -or $terminalQuote[1] -cne 'a"b') {throw 'A representable terminal literal quote was changed'}
|
||||
# Direct native control, deliberately NOT passed through our representability gate.
|
||||
# A literal unmatched quote swallows the following argument; doubling/backslashes
|
||||
# are not a batch equivalent of CRT quote escaping. The fixture performs no action.
|
||||
$native=[ordered]@{
|
||||
file=(Join-Path $env:WINDIR 'System32\cmd.exe')
|
||||
command_line='/d /s /v:off /c ""%OTCHE_NATIVE_PATH%" %OTCHE_NATIVE_ARG% "tail""'
|
||||
environment=@{OTCHE_NATIVE_PATH=$batch;OTCHE_NATIVE_ARG='"a"b"'}
|
||||
}
|
||||
$nativeValues=@(Invoke-Fixture $native)
|
||||
if ($nativeValues[0] -ceq 'a"b' -and $nativeValues[1] -ceq 'tail') {throw 'Native quote-control assumption changed; revisit representability validation'}
|
||||
[ordered]@{native_unmatched_quote_control=@($nativeValues[0],$nativeValues[1]);requested=@('a"b','tail')} | ConvertTo-Json -Compress
|
||||
$native.environment.OTCHE_NATIVE_ARG='"say "two words""'
|
||||
$nativeWords=@(Invoke-Fixture $native)
|
||||
if ($nativeWords[0] -ceq 'say "two words"' -and $nativeWords[1] -ceq 'tail') {throw 'Native inner-quote separator assumption changed; revisit validation'}
|
||||
[ordered]@{native_inner_quote_separator_control=@($nativeWords[0],$nativeWords[1],$nativeWords[2]);requested=@('say "two words"','tail')} | ConvertTo-Json -Compress
|
||||
foreach ($value in @('a"b','"literal','say "two words"',"one`ntwo")) {
|
||||
$settings.args=@($value,'tail');$rejected=$false
|
||||
try {Get-OtcheLaunch $batch $settings $root | Out-Null} catch {$rejected=$_.Exception.Message.StartsWith('INCOMPATIBLE:')}
|
||||
if (!$rejected) {throw ('Unrepresentable batch value accepted: '+$value)}
|
||||
}
|
||||
$settings.args=@('/norestart','COMPANY=Acme "Widgets" & Co','EMPTY=','TRAIL=C:\space path\','/m','STATUS01','/n','{00000001-0002-0000-0000-624474736554}')
|
||||
$launch=Get-OtcheLaunch (Join-Path $root 'sample.msi') $settings $root
|
||||
if (!$launch.command_line.Contains('COMPANY="Acme ""Widgets"" & Co"') -or !$launch.command_line.Contains('EMPTY=""') -or !$launch.command_line.Contains('TRAIL="C:\space path\"')) {throw 'MSI documented property quoting changed'}
|
||||
foreach ($option in @('/norestart','/promptrestart','/forcerestart')) {$settings.args=@($option);$null=Get-OtcheLaunch (Join-Path $root 'sample.msi') $settings $root}
|
||||
foreach ($options in @(@('/x','other.msi'),@('/i','other.msi'),@('/log','other.log'),@('/quiet'),@('ACTION=ADMIN'))) {
|
||||
$settings.args=$options;$rejected=$false
|
||||
try {Get-OtcheLaunch (Join-Path $root 'sample.msi') $settings $root | Out-Null} catch {$rejected=$true}
|
||||
if (!$rejected) {throw 'MSI sample/operation/log/UI replacement accepted'}
|
||||
}
|
||||
'PASS actual CMD/BAT argument boundaries and native quote counterexample; MSI modifier/property construction and required operation/log/UI protections. No MSI executed.'
|
||||
} finally {
|
||||
if ([IO.Directory]::Exists($root)) {[IO.Directory]::Delete($root,$true)}
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
#requires -Version 5.1
|
||||
[CmdletBinding()]
|
||||
param()
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference='Stop'
|
||||
Import-Module "$PSScriptRoot\Otche.Common.psm1" -Force
|
||||
Initialize-OtcheGrub
|
||||
$root=Join-Path ([IO.Path]::GetTempPath()) ('otche-grub-test-'+[guid]::NewGuid().ToString('N'))
|
||||
[void][IO.Directory]::CreateDirectory($root)
|
||||
$writer=$null;$junction=$null;$originalAcl=$null
|
||||
try {
|
||||
foreach ($name in @('a','b','out')) {[void][IO.Directory]::CreateDirectory((Join-Path $root $name))}
|
||||
[IO.File]::WriteAllBytes("$root\a\same.log",[byte[]]@(0,1,255,13,10))
|
||||
[IO.File]::WriteAllText("$root\b\same.log",'second')
|
||||
[IO.File]::WriteAllBytes("$root\empty.log",[byte[]]@())
|
||||
[IO.File]::WriteAllText("$root\100%done{copy}.log",'literal')
|
||||
$writer=[IO.File]::Open("$root\a\same.log",[IO.FileMode]::Open,[IO.FileAccess]::ReadWrite,[IO.FileShare]::ReadWrite)
|
||||
$paths=@("$root\a\same.log","$root\b\same.log","$root\empty.log","$root\missing.log","$root\100%done{copy}.log")
|
||||
$manifest=[pscustomobject]@{settings=[pscustomobject]@{grub_paths=$paths};grub_limits=[pscustomobject]@{file_bytes=100;total_bytes=100;seconds=10}}
|
||||
$result=Get-OtcheGrubSnapshot $manifest "$root\out"
|
||||
if ($result.collected -ne 4 -or $result.status -ne 'partial') {throw ('Unexpected collection: '+($result|ConvertTo-Json -Depth 8 -Compress))}
|
||||
if ($result.files[0].status -ne 'collected') {throw 'An ordinary live writer wrongly prevented readable log collection'}
|
||||
if ($result.files[2].size -ne 0 -or $result.files[2].sha256 -ne 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855') {throw 'Empty-file evidence is wrong'}
|
||||
if ($result.files[3].status -ne 'missing' -or $null -ne $result.files[3].size -or $null -ne $result.files[3].sha256) {throw 'Missing-file metadata is dishonest'}
|
||||
if ([Convert]::ToBase64String([IO.File]::ReadAllBytes("$root\out\grub-001.bin")) -ne 'AAH/DQo=') {throw 'Captured active-writer bytes differ'}
|
||||
$writer.Dispose();$writer=$null
|
||||
foreach ($bad in @('\\server\share\file','\\?\C:\file','\\.\pipe\name','C:\a\..\b','C:\a\x:stream','C:\a\*','C:\a\NUL','C:\a\COM1.txt','C:relative','%TEMP%\file')) {
|
||||
$denied=$false;try {[Otche.Grub]::Validate($bad)} catch {$denied=$true};if (!$denied) {throw "Unsafe path accepted: $bad"}
|
||||
}
|
||||
$denied=$false;try {[Otche.Grub]::Copy("$root\a","$root\dir.bin",100,[DateTime]::UtcNow.AddSeconds(5))} catch {$denied=$true};if (!$denied) {throw 'Directory accepted'}
|
||||
$denied=$false;try {[Otche.Grub]::Copy("$root\b\same.log","$root\large.bin",2,[DateTime]::UtcNow.AddSeconds(5))} catch {$denied=$true};if (!$denied) {throw 'Byte limit ignored'}
|
||||
$junction=Join-Path $root 'junction'
|
||||
New-Item -ItemType Junction -Path $junction -Target "$root\a" | Out-Null
|
||||
$denied=$false;try {[Otche.Grub]::Copy("$junction\same.log","$root\link.bin",100,[DateTime]::UtcNow.AddSeconds(5))} catch {$denied=$true};if (!$denied) {throw 'Reparse path followed'}
|
||||
[IO.File]::WriteAllText("$root\denied.log",'protected fixture')
|
||||
$originalAcl=Get-Acl -LiteralPath "$root\denied.log"
|
||||
$acl=Get-Acl -LiteralPath "$root\denied.log"
|
||||
New-Item -ItemType HardLink -Path "$root\hard.log" -Target "$root\b\same.log" | Out-Null
|
||||
$denied=$false;try {[Otche.Grub]::Copy("$root\hard.log","$root\hard-export.bin",100,[DateTime]::UtcNow.AddSeconds(5))} catch {$denied=$_.Exception.GetBaseException().Message -like '*Hard-linked*'};if (!$denied) {throw 'Hardlinked staging file was not rejected before export'}
|
||||
$sid=[Security.Principal.WindowsIdentity]::GetCurrent().User
|
||||
$acl.AddAccessRule([Security.AccessControl.FileSystemAccessRule]::new($sid,'ReadData','Deny'))
|
||||
Set-Acl -LiteralPath "$root\denied.log" -AclObject $acl
|
||||
$manifest.settings.grub_paths=@("$root\denied.log")
|
||||
[void][IO.Directory]::CreateDirectory("$root\denied-out")
|
||||
$deniedResult=Get-OtcheGrubSnapshot $manifest "$root\denied-out"
|
||||
if ($deniedResult.files[0].status -ne 'access_denied' -or $null -ne $deniedResult.files[0].size) {throw 'Access denied was not reported honestly'}
|
||||
[ordered]@{passed=$true;active_writer_read=$true;same_basename=$true;empty_sha256=$true;missing_null_metadata=$true;literal_percent_braces=$true;unsafe_paths_denied=$true;directory_denied=$true;byte_limit=$true;reparse_denied=$true;hardlink_denied=$true;access_denied=$true} | ConvertTo-Json -Compress
|
||||
} finally {
|
||||
if ($null -ne $writer) {$writer.Dispose()}
|
||||
if ($null -ne $originalAcl) {Set-Acl -LiteralPath "$root\denied.log" -AclObject $originalAcl}
|
||||
if ($null -ne $junction -and [IO.Directory]::Exists($junction)) {[IO.Directory]::Delete($junction)}
|
||||
if ([IO.Directory]::Exists($root)) {[IO.Directory]::Delete($root,$true)}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
#requires -Version 5.1
|
||||
[CmdletBinding(DefaultParameterSetName='Ready')]
|
||||
param([Parameter(Mandatory=$true)][ValidateSet('user','admin')][string]$Privilege,
|
||||
[Parameter(ParameterSetName='Desktop')][switch]$DesktopProbe,
|
||||
[Parameter(ParameterSetName='Baseline')][switch]$BaselineOnly,
|
||||
[Parameter(ParameterSetName='Boot')][switch]$BootOnly)
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference='Stop'
|
||||
Import-Module "$PSScriptRoot\Otche.Common.psm1" -Force
|
||||
if ($BootOnly) { @{boot_id=(Get-OtcheBootId)}|ConvertTo-Json -Compress;exit 0 }
|
||||
if ($DesktopProbe) {
|
||||
$request=Read-OtcheJson 'C:\ProgramData\Otche\control\desktop-probe.json' 4096
|
||||
if ([string]$request.nonce -notmatch '^[0-9a-f]{32}$' -or [DateTime]::UtcNow -gt [DateTime]::Parse($request.deadline) -or [DateTime]::Parse($request.deadline) -gt [DateTime]::UtcNow.AddSeconds(15)) {throw 'Desktop probe is invalid or expired'}
|
||||
$probe=[ordered]@{nonce=$request.nonce;boot_id=(Get-OtcheBootId);session_id=0;user='';ready=$false;error=''}
|
||||
try {$session=Get-OtcheSession 'user' -Current;$probe.session_id=$session.session_id;$probe.user=$session.user;$probe.ready=$true} catch {$probe.error=$_.Exception.Message}
|
||||
Write-OtcheJson 'C:\ProgramData\Otche\results\desktop-probe.json' $probe
|
||||
exit 0
|
||||
}
|
||||
$result=[ordered]@{ready=$false;session_id=$null;user='';privilege=$Privilege;boot_id=$null;baseline=$null;defender=$null;environment=$null;error=''}
|
||||
try {
|
||||
$session=Get-OtcheSession $Privilege
|
||||
$result.session_id=$session.session_id; $result.user=$session.user; $result.boot_id=Get-OtcheBootId
|
||||
$baseline=Get-OtcheBaseline
|
||||
$result.baseline=@{fingerprint=$baseline.fingerprint;qualified=$baseline.qualified;errors=$baseline.errors}
|
||||
$result.defender=$baseline.defender; $result.environment=$baseline.environment
|
||||
$task=Get-ScheduledTask -TaskName ('Otche-'+$Privilege) -ErrorAction Stop
|
||||
$source=Read-OtcheJson 'C:\ProgramData\Otche\source.json'
|
||||
$taskSid=$(if ($task.Principal.UserId -match '^S-1-') {([Security.Principal.SecurityIdentifier]::new($task.Principal.UserId)).Value} else {([Security.Principal.NTAccount]::new($task.Principal.UserId)).Translate([Security.Principal.SecurityIdentifier]).Value})
|
||||
if ($taskSid -ne $source.account_sid -or [string]$task.Principal.LogonType -ne 'Interactive' -or [string]$task.Principal.RunLevel -ne $(if ($Privilege -eq 'admin') {'Highest'} else {'Limited'})) {throw 'Prepared InteractiveToken task principal has drifted'}
|
||||
if ([string]$task.State -eq 'Disabled') {throw 'Prepared runner task is disabled'}
|
||||
$collector=Get-ScheduledTask -TaskName 'Otche-Collector' -ErrorAction Stop
|
||||
if ([string]$collector.Principal.UserId -notin @('SYSTEM','S-1-5-18') -or [string]$collector.State -eq 'Disabled') {throw 'SYSTEM telemetry collector is not prepared'}
|
||||
$exe="$env:WINDIR\System32\WindowsPowerShell\v1.0\powershell.exe"
|
||||
$runnerArguments='-NoProfile -NonInteractive -File "C:\ProgramData\Otche\runner\Invoke-OtcheDispatch.ps1" -Privilege '+$Privilege
|
||||
$collectorArguments='-NoProfile -NonInteractive -File "C:\ProgramData\Otche\runner\Collect-Otche.ps1"'
|
||||
if (@($task.Actions).Count -ne 1 -or $task.Actions[0].Execute -ine $exe -or $task.Actions[0].Arguments -cne $runnerArguments) {throw 'Prepared runner action has drifted'}
|
||||
if (@($collector.Actions).Count -ne 1 -or $collector.Actions[0].Execute -ine $exe -or $collector.Actions[0].Arguments -cne $collectorArguments) {throw 'Prepared collector action has drifted'}
|
||||
if (!$BaselineOnly) {
|
||||
$desktopTask=Get-ScheduledTask -TaskName 'Otche-DesktopReady' -ErrorAction Stop
|
||||
$desktopSid=$(if ($desktopTask.Principal.UserId -match '^S-1-') {$desktopTask.Principal.UserId} else {([Security.Principal.NTAccount]::new($desktopTask.Principal.UserId)).Translate([Security.Principal.SecurityIdentifier]).Value})
|
||||
$desktopArguments='-NoProfile -NonInteractive -WindowStyle Hidden -File "C:\ProgramData\Otche\runner\Test-OtcheReady.ps1" -Privilege user -DesktopProbe'
|
||||
if ($desktopSid -ne $source.account_sid -or [string]$desktopTask.Principal.LogonType -ne 'Interactive' -or [string]$desktopTask.Principal.RunLevel -ne 'Limited' -or [string]$desktopTask.State -ne 'Ready' -or @($desktopTask.Actions).Count -ne 1 -or $desktopTask.Actions[0].Execute -ine $exe -or $desktopTask.Actions[0].Arguments -cne $desktopArguments) {throw 'Prepared desktop readiness task has drifted or is busy'}
|
||||
$nonce=[guid]::NewGuid().ToString('N');$deadline=[DateTime]::UtcNow.AddSeconds(12)
|
||||
Write-OtcheJson 'C:\ProgramData\Otche\control\desktop-probe.json' @{nonce=$nonce;deadline=$deadline.ToString('o')}
|
||||
Start-ScheduledTask -TaskName 'Otche-DesktopReady'
|
||||
$probe=$null
|
||||
do {
|
||||
Start-Sleep -Milliseconds 200
|
||||
try {$candidate=Read-OtcheJson 'C:\ProgramData\Otche\results\desktop-probe.json' 4096;if ($candidate.nonce -ceq $nonce) {$probe=$candidate;break}} catch {}
|
||||
} while ([DateTime]::UtcNow -lt $deadline)
|
||||
if ($null -eq $probe) {throw 'Fresh interactive desktop probe timed out'}
|
||||
if (!$probe.ready -or [DateTime]::UtcNow -gt $deadline) {throw ('Interactive desktop unavailable or probe expired: '+$probe.error)}
|
||||
if ($probe.boot_id -cne $result.boot_id -or $probe.session_id -ne $session.session_id -or $probe.user -cne $session.user) {throw 'Desktop probe boot/account/session identity mismatch'}
|
||||
}
|
||||
$result.ready=$baseline.qualified
|
||||
if (!$result.ready) {$result.error=$baseline.errors -join '; '}
|
||||
} catch {$result.error=$_.Exception.Message}
|
||||
$result | ConvertTo-Json -Depth 12 -Compress
|
||||
if (!$result.ready) {exit 1}
|
||||
Reference in New Issue
Block a user