omarandClaude Opus 5
625942834b
The line naming a nonzero `go test` status was printed only when every privileged test had produced a verdict — on the reasoning that a named FAILED already explains the status. The case that actually happens is the opposite one: the run dies at package level, so it names no test, so the loop above prints MISSING for all of them, and the one line pointing at the real cause was the one suppressed. A reader then goes hunting for three vanished tests instead of at the build error above. To be exact about what was and was not broken, because the framing matters: the exit status was never SWALLOWED. priv_bad is set by the MISSING branch, so FAILED is set and the gate fails either way — this was a diagnosis bug, not a correctness one. What changes is whether the log says why. Verified on the branch a green run never reaches, by driving the edited block with all four (priv_rc, priv_bad) combinations: the new message appears only for (1,1), the old one only for (1,0), and priv_bad/FAILED come out 1 in both. The full gate is green with the change in, which covers the (0,0) path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BHw89tdWddzhjUc4bAH4tS
878 lines
44 KiB
Bash
878 lines
44 KiB
Bash
#!/usr/bin/env bash
|
|
#
|
|
# run-tests.sh — THE test gate of the release tract.
|
|
#
|
|
# WHY THIS EXISTS (2026-07-26)
|
|
# Until now the release tract ran almost no tests. The only `go test` calls in
|
|
# the whole publishing path were scripts/build-shaterd.sh's one-package
|
|
# buildtags check and the three named tests scripts/check-router-tags.sh runs.
|
|
# The upstream .github/workflows/test.yml triggers on `stable`/`testing`/
|
|
# `unstable` — branches this fork does not have — and Gitea does not read
|
|
# .github/workflows at all once .gitea/workflows exists. Net effect: 115 of the
|
|
# 116 test files under shater/** had never executed in CI, and
|
|
# TestDNSFilterRemoteBlocklistHTTPClient shipped red through two releases
|
|
# before anyone ran it by hand.
|
|
#
|
|
# WHAT IT GUARANTEES
|
|
# 1. The suite runs under the SHIPPED build tags (scripts/router-tags.sh), not
|
|
# under some CI-local tag set. This is not cosmetic: the AmneziaWG tests in
|
|
# transport/wireguard are `//go:build with_awg` — 1 test file compiles
|
|
# without the tag set, 7 with it. The 2026-07-25 WireGuard outage was
|
|
# exactly a "built with X, verified with Y" gap.
|
|
# 2. It runs on linux. shater/generate has 44 test files on linux against 32 on
|
|
# windows/darwin; the linux-only half is where the routing, ruleset, DNS and
|
|
# health tests live.
|
|
# 3. Nothing is skipped SILENTLY. Six machine checks:
|
|
# - the tag set may only ADD test files, never hide them (a test behind
|
|
# `//go:build !with_awg` would vanish from the gate — this fails first);
|
|
# - every package that has tests must report `ok` by name; a suite that
|
|
# compiles down to "no test files" fails the gate instead of passing it;
|
|
# - every ORDINARY test that calls t.Skip is named in the output and must
|
|
# be DECLARED in SKIP_DECLARED below with the reason it cannot run here;
|
|
# an undeclared skip fails the gate. This is why the suites run with -v:
|
|
# without it a skipped test prints nothing whatsoever and the package
|
|
# still reports `ok`. It was not a hypothetical — shater/apply's
|
|
# TestApplyInstallsHoldWhenEngineFailsToStart, the W5 regression for
|
|
# "the engine died, the LAN must not be left open", guarded itself with
|
|
# a t.Skip whose condition had become permanently true, so it asserted
|
|
# nothing at all while the gate reported `ok shater/apply`;
|
|
# - every ^TestIntegration under the fork's trees must produce a verdict
|
|
# BY NAME ([5/7]). `ok <pkg>` is printed whether the privileged tests in
|
|
# that package ran or called t.Skip, so the second check cannot see them
|
|
# — and the gate would keep saying "passes every test we own" while the
|
|
# tests that need a real kernel never executed;
|
|
# - every NON-GO test file in the tree must be claimed by a named runner
|
|
# ([6/7]). The four checks above are all built on `go list`/`go test`, so
|
|
# a test in another language is invisible to them BY CONSTRUCTION — and
|
|
# that is not hypothetical either: openwrt/luci-app-shater/tests/
|
|
# status-readout.test.js, 24 assertions over the one screen an operator
|
|
# reaches while the LAN is cut off, was run by nothing at all;
|
|
# - the non-Go suites this gate owns produce a verdict BY NAME ([7/7]),
|
|
# including "did not run: no node here", which then replaces the closing
|
|
# banner.
|
|
# A guard that silently runs nothing is worse than no guard (same rule as
|
|
# scripts/check-router-tags.sh).
|
|
#
|
|
# WHAT IT DOES NOT SEE, AND WHAT DOES (2026-07-27)
|
|
# Everything above is about whether a test RAN and what it ASSERTED. None of it
|
|
# can see what a test DID to the machine. shater/stats/store_test.go ended with
|
|
# `os.Remove(statsFilePath())` — the PRODUCT path — so every run of this gate on
|
|
# the testbed or the router deleted /etc/shater/stats.db, and every instrument
|
|
# here reported `ok shater/stats`. Six packages were doing something of the kind.
|
|
# Two instruments now cover it, and neither is inside the steps below:
|
|
# - shater/testguard/fsisolation_test.go — a Go test that reads the SOURCE of
|
|
# every _test.go under shater/ and fails BY NAME when a filesystem-mutating
|
|
# call is handed a path that is not provably a temp dir. It runs as part of
|
|
# [2/7] and [4/7] like any other test, so it needs no step of its own; what
|
|
# it cannot see is damage done by PRODUCT code that a test merely calls.
|
|
# - scripts/check-test-fs-isolation.sh — the dynamic half, for exactly that
|
|
# blind spot: it seeds a router-shaped canary tree in a container, runs this
|
|
# whole suite, and diffs. NOT run from here on purpose — it costs a second
|
|
# full suite, and its method (let the damage happen, then look) must never
|
|
# be pointed at a real /etc. Run it by hand when tests touch anything that
|
|
# resolves a product path.
|
|
#
|
|
# Usage:
|
|
# scripts/run-tests.sh # full gate (~3 min warm on the runner)
|
|
# scripts/run-tests.sh --no-race # skip the -race pass (faster; local loop)
|
|
#
|
|
# Env:
|
|
# SHATER_GO_IMAGE docker image used to reach linux from a non-linux host
|
|
# (default golang:1.26 — keep it >= go.mod's toolchain).
|
|
# SHATER_NO_DOCKER=1 fail instead of falling back to docker.
|
|
# SHATER_REQUIRE_PRIVILEGED=1
|
|
# turn [5/7]'s "did not run here" report into a hard
|
|
# failure. Use it on the OpenWrt VM or in any pre-release
|
|
# run that must actually have exercised the kernel paths.
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
cd "$REPO"
|
|
|
|
RACE=1
|
|
for a in "$@"; do
|
|
case "$a" in
|
|
--no-race) RACE=0 ;;
|
|
-h|--help) sed -n '2,67p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
|
*) echo "run-tests: unknown flag: $a" >&2; exit 2 ;;
|
|
esac
|
|
done
|
|
|
|
# shellcheck source=router-tags.sh
|
|
. "$SCRIPT_DIR/router-tags.sh"
|
|
|
|
# The fork's own trees plus the upstream trees the fork edits (adapter/, route/,
|
|
# option/, dns/ all carry shater changes). ROOTS, not a hand-kept package list: a
|
|
# new package with tests joins the gate the moment it is created, which is the
|
|
# whole point.
|
|
ROOTS=(./shater/... ./protocol/... ./transport/... ./adapter/... ./route/... ./option/... ./dns/...)
|
|
|
|
# common/ is mostly upstream, but common/tls, common/tlsfragment, common/sniff
|
|
# and common/urltest carry fork behaviour (D13 DPI-bypass, urltest health), so it
|
|
# is in — minus the privileged integration tests below.
|
|
ROOTS_COMMON=(./common/...)
|
|
# SKIP, WITH REASON: common/tlsspoof's TestIntegration* enter TCP_REPAIR and
|
|
# need CAP_NET_ADMIN. The act_runner job container runs as root but WITHOUT
|
|
# that capability, so they do not skip — they FAIL. Excluded by name so the
|
|
# rest of common/ can be a real gate instead of a permanently red one. (On
|
|
# linux every tlsspoof test is a TestIntegration*, so that package is
|
|
# effectively uncovered here; it is covered by the VM runs.)
|
|
# The ^TestIntegration prefix is the fork-wide marker for "needs capabilities
|
|
# the ordinary gate lacks", and [5/7] below leans on the same convention to
|
|
# catch privileged tests inside ROOTS, which are NOT name-filtered and would
|
|
# otherwise skip behind a green `ok <pkg>`. ROOTS_COMMON stays out of [5/7]:
|
|
# these fail rather than skip without the capability, and that is a decision
|
|
# about upstream code, not about the fork's own coverage.
|
|
SKIP_COMMON='^TestIntegration'
|
|
|
|
# SKIP, WITH REASON: the first -race run over this tree (2026-07-26 — nobody
|
|
# had ever run one) turned up two failures. One was a REAL product race:
|
|
# ClientBind.connect() touched its fields from both the Send() path and
|
|
# RoutineReceiveIncoming() with no lock, caught by
|
|
# transport/wireguard.TestAwgDetourClientBindDelivers; that one has since been
|
|
# fixed in client_bind.go and is NOT skipped — it is exactly what this pass is
|
|
# for. What was left:
|
|
# - shater/alert.TestExpiryDedupWithinDay — the test's own closure read a
|
|
# variable the test body wrote while Notifier.dispatch's goroutine was
|
|
# still delivering. FIXED 2026-07-26 (the simulated clock now has a mutex),
|
|
# so the entry is gone and the -race pass covers the whole tree again.
|
|
# Nothing is skipped under -race any more. Keep it that way: an entry here is a
|
|
# hole in the gate, so add one only with a named reason and delete it the moment
|
|
# the race is fixed.
|
|
RACE_SKIP='^$'
|
|
|
|
# --- the per-package deadline ------------------------------------------------
|
|
# GOTIMEOUT is `go test`'s own default, 10m. It is WRITTEN DOWN rather than
|
|
# inherited because on 2026-07-26 this gate reported a failure that did not
|
|
# exist: [4/7] printed
|
|
#
|
|
# FAIL shater/netplane 600.019s
|
|
# panic: test timed out after 10m0s
|
|
# running tests: TestApplyIfaceSysctlsCoversRuleDivertedIface
|
|
#
|
|
# over code that was neither hung nor wrong. Under -race that package really did
|
|
# need 663.8 s (measured 2026-07-27, golang:1.26, 32 cores) against 2.0 s without
|
|
# it — a 337x factor that no amount of "-race is slower" explains.
|
|
#
|
|
# THE ANSWER WAS NOT A BIGGER NUMBER, and this comment exists so nobody reaches
|
|
# for one next time. The 664 seconds were not work: nine of netplane's test files
|
|
# intercept nft/ip/ubus/uci/sysctl by RE-EXEC'ing the test binary as a no-op
|
|
# helper (the standard os/exec trick), and ThreadSanitizer sleeps
|
|
# `atexit_sleep_ms` — DEFAULT 1000 — before every -race process exits. ~660
|
|
# intercepted commands, one wall-clock second each, zero CPU; the per-test times
|
|
# came out as near-exact integers (12.17 s, 14.17 s, 129.62 s) because that is
|
|
# what they were counting. The children now run with GORACE=atexit_sleep_ms=0
|
|
# (shater/netplane/racehelperenv_test.go, which explains what that does and — more
|
|
# to the point — what it does NOT cost) and the package is 10.6 s under -race.
|
|
# shater/devices had the same disease for the same reason: 0.108 s plain,
|
|
# 28.412 s under -race, 0.358 s once its children stopped sleeping.
|
|
#
|
|
# So the deadline stays where it was, and it stays there as a HANG DETECTOR.
|
|
# Measured by this script on 2026-07-27 after both fixes, [4/7] end to end: 56 s,
|
|
# slowest package shater/generate at 18.9 s — 10m is ~32x that. If this fires
|
|
# again the tests are BLOCKED, not slow: read the goroutine dump the panic prints
|
|
# and fix the block. Raising it is only ever an answer with a fresh measurement
|
|
# written down beside it, because a deadline raised past a real hang stops being
|
|
# a deadline.
|
|
GOTIMEOUT=10m
|
|
|
|
# --- the ONLY skips this gate accepts ----------------------------------------
|
|
# A t.Skip is invisible to every other check here: the test binary exits 0, the
|
|
# package prints `ok <pkg>`, and the name of the test that did not run appears
|
|
# NOWHERE unless -v is on. That is how shater/apply's W5 regression —
|
|
# TestApplyInstallsHoldWhenEngineFailsToStart, the only END-TO-END test between
|
|
# "the engine died" and "the LAN forwards to the WAN in the clear" — came to
|
|
# assert nothing at all: it broke the engine by pointing a rule-set at
|
|
# /nonexistent/nope.srs and stood itself down with t.Skip when that failed to
|
|
# break anything, and it stopped breaking anything once LocalRuleSet.reloadFile
|
|
# began treating an unreadable file as an empty one. Measured 2026-07-26 in
|
|
# golang:1.26: the skip fired unconditionally, and the package still printed
|
|
# `ok shater/apply`.
|
|
#
|
|
# So the suites below run with -v and every `--- SKIP` is matched against this
|
|
# list. A skip that is not here fails the gate BY NAME. Skips that genuinely
|
|
# cannot run in some environment are not forbidden — they are DECLARED, with the
|
|
# reason, and printed on every run so nobody mistakes the gate's silence for
|
|
# coverage.
|
|
#
|
|
# Format: '<extended regexp matched against the full test name>|<reason>'.
|
|
# The reason is shown verbatim next to the test on every run; write it for
|
|
# someone deciding whether the gate proved what they think it proved.
|
|
SKIP_DECLARED=(
|
|
'^TestIntegration|privileged: needs root + CAP_NET_ADMIN + /dev/net/tun. NOT waved through — [5/7] below gives every one of these a verdict by name, and an unrunnable one REPLACES the closing banner so this run cannot claim it covered them.'
|
|
'^TestCompiledTagsMatchTheShippedSet$|compares the tags COMPILED INTO a binary with router-tags.sh, and needs the harness that builds that binary and sets SHATER_ROUTER_TAG_CHECK=1. The harness is scripts/check-router-tags.sh, which the release tract runs separately; here there is no such binary to read.'
|
|
)
|
|
|
|
# --- every NON-GO test file must be claimed by a runner ----------------------
|
|
# The Go half of this gate cannot see a test written in another language, and the
|
|
# review of 2026-07-26 found what that costs: openwrt/luci-app-shater/tests/
|
|
# status-readout.test.js — 236 lines, six recorded fixtures, 24 assertions, the
|
|
# only thing checking what the LuCI dashboard tells an operator while the engine
|
|
# is down — was executed by NOTHING. Not by panel/package.json's `test` script
|
|
# (`node --test src/*.test.ts`, panel/src only), not by scripts/run-panel-tests.sh
|
|
# (same glob), not by any step here. And [1/7] could not report it, because
|
|
# `go list` is the instrument and a .js file is invisible to it BY CONSTRUCTION.
|
|
#
|
|
# So [6/7] enumerates the tree's non-Go test files and requires each to be claimed
|
|
# by a runner named HERE. A new .test.js/.test.ts/.spec.*/test_*.py that no runner
|
|
# picks up fails the gate by name on the day it is committed, instead of sitting
|
|
# there looking like coverage.
|
|
#
|
|
# Format: '<extended regexp matched against the repo-relative path>|<runner>'.
|
|
# Positive and CLOSED on purpose: a file that matches nothing is a failure, not a
|
|
# default. The Go files are deliberately NOT in scope — `_test.go` under the
|
|
# declared ROOTS is what [1/7]+[2/7] already prove ran, and pulling the rest of
|
|
# the upstream tree in here would be a different decision.
|
|
NONGO_TEST_RUNNERS=(
|
|
'^panel/src/[^/]+\.test\.ts$|scripts/run-panel-tests.sh (node --test via panel/package.json); CI runs it as its own step on node 24, before this script'
|
|
'^openwrt/luci-app-shater/tests/[^/]+\.test\.js$|[7/7] of this script'
|
|
)
|
|
|
|
# The non-Go suites [7/7] RUNS, as `node <file>`. panel/src is not here: it has its
|
|
# own script with its own npm install, and duplicating it would mean two places to
|
|
# keep right. Each entry is a glob; a glob that matches nothing is a failure (a
|
|
# renamed file must be reported as that, not as a fast green step).
|
|
JS_SUITES=('openwrt/luci-app-shater/tests/*.test.js')
|
|
|
|
# JS_UNVERIFIED collects the suites that did NOT run, by path, for the closing
|
|
# banner — the same treatment PRIV_UNVERIFIED gets, and for the same reason.
|
|
JS_UNVERIFIED=""
|
|
|
|
# js_step runs the declared non-Go suites and prints a verdict for each BY NAME.
|
|
# Defined up here because it is called from TWO places: the non-linux re-exec
|
|
# below runs it on the HOST, where node usually exists, rather than let the
|
|
# golang image (which has none) report "did not run" on every single local run —
|
|
# a banner that always fires is a banner nobody reads.
|
|
#
|
|
# `node <file>`: these are standalone harnesses that exit non-zero on a failed
|
|
# assertion, not `node --test` modules.
|
|
#
|
|
# KNOWN LIMIT, stated rather than papered over: the verdict is the process exit
|
|
# code. A harness gutted of its assertions that still exits 0 reads as a pass —
|
|
# the same limit scripts/run-panel-tests.sh already names for `node --test`.
|
|
# Deletion, rename, a throw and a failed assertion are all caught.
|
|
#
|
|
# Returns 1 if a suite failed or the globs matched nothing. Sets JS_UNVERIFIED
|
|
# when there is no node to run them with.
|
|
js_step() { # $1 = where we are, in words, for the "no node" line
|
|
local where="$1" f g rc bad=0 tmp
|
|
local files=()
|
|
shopt -s nullglob
|
|
for g in "${JS_SUITES[@]}"; do
|
|
files+=($g)
|
|
done
|
|
shopt -u nullglob
|
|
if [ "${#files[@]}" -eq 0 ]; then
|
|
echo " FAILED [js]: JS_SUITES matched no file at all. Either the glob is wrong or" >&2
|
|
echo " the suite was renamed/deleted — both must be said, not passed over." >&2
|
|
return 1
|
|
fi
|
|
if ! command -v node >/dev/null 2>&1; then
|
|
echo " node: NOT AVAILABLE $where — these suites did NOT run:"
|
|
for f in "${files[@]}"; do
|
|
echo " DID NOT RUN $f"
|
|
JS_UNVERIFIED="$JS_UNVERIFIED $f"
|
|
done
|
|
return 0
|
|
fi
|
|
echo " node: $(node --version) $where, ${#files[@]} suite(s)"
|
|
tmp="$(mktemp)"
|
|
for f in "${files[@]}"; do
|
|
set +e
|
|
node "$f" >"$tmp" 2>&1
|
|
rc=$?
|
|
set -e
|
|
if [ "$rc" -eq 0 ]; then
|
|
echo " RAN $f"
|
|
else
|
|
echo " FAILED $f (exit $rc)" >&2
|
|
sed 's/^/ | /' "$tmp" >&2
|
|
bad=1
|
|
fi
|
|
done
|
|
rm -f "$tmp"
|
|
return "$bad"
|
|
}
|
|
|
|
echo "== shater test gate =="
|
|
echo " tags : $SHATER_ROUTER_TAGS"
|
|
echo " ldflags: $SHATER_ROUTER_LDFLAGS"
|
|
echo " race : $([ "$RACE" -eq 1 ] && echo yes || echo no)"
|
|
echo
|
|
|
|
# --- linux, or re-exec on linux ---------------------------------------------
|
|
# The linux-only half of the suite is the half worth running (see header). From a
|
|
# non-linux host, re-exec inside a golang container rather than quietly testing
|
|
# 32 of shater/generate's 44 files — a partial gate reads exactly like a passing
|
|
# one.
|
|
if [ "$(go env GOOS)" != "linux" ] && [ "${SHATER_TESTS_IN_DOCKER:-0}" != "1" ]; then
|
|
if [ "${SHATER_NO_DOCKER:-0}" = "1" ] || ! command -v docker >/dev/null 2>&1; then
|
|
echo " ERROR: the gate needs linux (GOOS=$(go env GOOS)) and docker is unavailable/disabled." >&2
|
|
echo " Run it on the linux CI runner or the OpenWrt VM." >&2
|
|
exit 1
|
|
fi
|
|
image="${SHATER_GO_IMAGE:-golang:1.26}"
|
|
echo "== re-exec on linux via docker ($image) =="
|
|
host_repo="$REPO"
|
|
command -v cygpath >/dev/null 2>&1 && host_repo="$(cygpath -w "$REPO")"
|
|
# Hand the container CAP_NET_ADMIN and /dev/net/tun when this host's docker
|
|
# can. shater/generate's ^TestIntegration tests open a real TUN and stand a
|
|
# real engine on it; without the device they skip, and a dev running the gate
|
|
# by hand would get a green result that never touched the kernel path the
|
|
# branch is about. The dev host CAN give them (Docker Desktop's VM has the tun
|
|
# module) — the CI runner cannot, which is what [5/7] exists to say out loud.
|
|
# PROBED, never assumed: a docker whose kernel lacks tun refuses --device and
|
|
# would take the whole gate down with it.
|
|
priv_flags=()
|
|
if MSYS2_ARG_CONV_EXCL='*' MSYS_NO_PATHCONV=1 docker run --rm \
|
|
--cap-add NET_ADMIN --device /dev/net/tun "$image" true >/dev/null 2>&1; then
|
|
priv_flags=(--cap-add NET_ADMIN --device /dev/net/tun)
|
|
echo " CAP_NET_ADMIN + /dev/net/tun: available — the privileged tests will really run"
|
|
else
|
|
echo " CAP_NET_ADMIN + /dev/net/tun: NOT available from this docker — [5/7] will report the gap"
|
|
fi
|
|
# The non-Go suites do not need linux, and this host very likely has node while
|
|
# the golang image certainly does not. Run them HERE, so the local loop really
|
|
# executes them instead of being told every single time that it did not: a
|
|
# banner that always fires is a banner nobody reads, and that is how a report
|
|
# stops being a report. Their verdict is folded into this script's exit status
|
|
# below, and the container is told not to repeat them.
|
|
host_js_rc=0
|
|
js_flags=()
|
|
if command -v node >/dev/null 2>&1; then
|
|
echo "== [7/7] the non-Go suites (node), run on this host before the re-exec =="
|
|
js_step "on this host ($image has none)" || host_js_rc=1
|
|
js_flags=(-e SHATER_JS_ALREADY_RAN=1)
|
|
echo
|
|
fi
|
|
MSYS2_ARG_CONV_EXCL='*' MSYS_NO_PATHCONV=1 docker run --rm \
|
|
"${priv_flags[@]+"${priv_flags[@]}"}" \
|
|
"${js_flags[@]+"${js_flags[@]}"}" \
|
|
-v "$host_repo":/src \
|
|
-v shater-tagcheck-gomod:/go/pkg/mod \
|
|
-v shater-tagcheck-gocache:/root/.cache/go-build \
|
|
-w /src \
|
|
-e SHATER_TESTS_IN_DOCKER=1 \
|
|
-e SHATER_REQUIRE_PRIVILEGED="${SHATER_REQUIRE_PRIVILEGED:-0}" \
|
|
"$image" bash -c '
|
|
# netplane.L3SlotFor asks the kernel through `ip link show` and reclaims a
|
|
# stale slot through `ip link del`. Without iproute2 EVERY slot reads as
|
|
# free, so TestIntegrationL3StaleSlotIsReclaimed refuses to run rather than
|
|
# pass while proving the opposite of what it claims — and [5/7] then fails
|
|
# the whole gate, correctly. golang:1.26 ships no iproute2, so install it
|
|
# here rather than let the image quietly narrow what this gate can verify.
|
|
# On a Linux host the script never re-execs, and the router has ip-full as
|
|
# a hard dependency, so this is the docker path only.
|
|
if ! command -v ip >/dev/null 2>&1; then
|
|
echo " iproute2: absent from '"$image"' — installing (the slot-reclaim test needs it)"
|
|
apt-get update -qq >/dev/null 2>&1 && apt-get install -y -qq iproute2 >/dev/null 2>&1 \
|
|
|| echo " iproute2: INSTALL FAILED — [5/7] will report the gap by name"
|
|
fi
|
|
exec bash scripts/run-tests.sh "$@"
|
|
' _ "$@"
|
|
docker_rc=$?
|
|
if [ "$host_js_rc" -ne 0 ]; then
|
|
echo "== TEST GATE FAILED — a non-Go suite failed on the host (see [7/7] above). ==" >&2
|
|
exit 1
|
|
fi
|
|
exit "$docker_rc"
|
|
fi
|
|
|
|
ALL_ROOTS=("${ROOTS[@]}" "${ROOTS_COMMON[@]}")
|
|
|
|
# --- [1/4] the tag set may only ADD test files, never hide them --------------
|
|
# `go list` counts the test files the compiler would actually take. If adding the
|
|
# shipped tags REMOVES a test file from any package, that test exists but the
|
|
# gate would never see it — which is the failure mode this whole script is about,
|
|
# just pointed the other way.
|
|
echo "== [1/7] no test file is hidden by the shipped tag set =="
|
|
LISTFMT='{{.ImportPath}} {{len .TestGoFiles}} {{len .XTestGoFiles}}'
|
|
plain="$(go list -f "$LISTFMT" "${ALL_ROOTS[@]}")"
|
|
tagged="$(go list -tags "$SHATER_ROUTER_TAGS" -f "$LISTFMT" "${ALL_ROOTS[@]}")"
|
|
hidden=0
|
|
while read -r pkg t x; do
|
|
[ -n "${pkg:-}" ] || continue
|
|
n_plain=$((t + x))
|
|
[ "$n_plain" -gt 0 ] || continue
|
|
line="$(awk -v p="$pkg" '$1 == p { print; exit }' <<<"$tagged")"
|
|
if [ -z "$line" ]; then
|
|
echo " HIDDEN: $pkg has $n_plain test file(s) untagged but no package at all under the shipped tags" >&2
|
|
hidden=1
|
|
continue
|
|
fi
|
|
read -r _ tt tx <<<"$line"
|
|
n_tagged=$((tt + tx))
|
|
if [ "$n_tagged" -lt "$n_plain" ]; then
|
|
echo " HIDDEN: $pkg — $n_plain test file(s) untagged, only $n_tagged under the shipped tags" >&2
|
|
hidden=1
|
|
elif [ "$n_tagged" -gt "$n_plain" ]; then
|
|
echo " +$((n_tagged - n_plain)) tag-gated test file(s): $pkg ($n_plain -> $n_tagged)"
|
|
fi
|
|
done <<<"$plain"
|
|
if [ "$hidden" -ne 0 ]; then
|
|
echo >&2
|
|
echo " FAILED: a test file is invisible to the tag set we ship. Either the" >&2
|
|
echo " constraint is wrong or the tag set is — do not paper over it" >&2
|
|
echo " by testing with different tags than we build with." >&2
|
|
exit 1
|
|
fi
|
|
echo
|
|
|
|
# --- the runner --------------------------------------------------------------
|
|
# Runs one suite and then PROVES it ran, at BOTH granularities:
|
|
# - package: every package `go list` says has tests must appear as `ok <pkg>`.
|
|
# `go test` over a package whose tests all vanished behind a build constraint
|
|
# prints "[no test files]" and exits 0 — a green run that verified nothing.
|
|
# - test: every `--- SKIP` must be declared in SKIP_DECLARED (check_skips).
|
|
# `ok <pkg>` is printed whether the tests inside ran or stood themselves down.
|
|
LOG="$(mktemp)"
|
|
trap 'rm -f "$LOG"' EXIT
|
|
FAILED=0
|
|
|
|
# check_skips reads the per-test verdicts of the suite in $LOG and refuses to let
|
|
# a t.Skip through unnamed. Returns non-zero on an undeclared skip.
|
|
check_skips() { # $1=label ; reads $LOG
|
|
local label="$1" name reason matched entry re bad=0
|
|
|
|
# THE CONTROL, and it comes first on purpose. Everything below reads `--- SKIP`
|
|
# lines, which exist only under `go test -v`. Drop the -v and this function
|
|
# reports a clean bill of health over a suite that skipped every test it had —
|
|
# a check against silent skipping that is itself silently skipping, which is
|
|
# exactly how the first cut of [5/7] shipped (`go test -list` failing to link,
|
|
# swallowed by `|| true`, reporting "none declared"). `=== RUN` is printed for
|
|
# every test the binary starts, so its absence means the verdicts are not being
|
|
# produced at all and this instrument is reading a blank page.
|
|
if ! grep -q '^=== RUN ' "$LOG"; then
|
|
echo " FAILED [$label]: not one '=== RUN' line in the output — per-test verdicts are" >&2
|
|
echo " not being produced (is -v still on?), so the skip check was reading a" >&2
|
|
echo " blank page and its silence means nothing." >&2
|
|
return 1
|
|
fi
|
|
|
|
while read -r name; do
|
|
[ -n "$name" ] || continue
|
|
matched=""
|
|
for entry in "${SKIP_DECLARED[@]}"; do
|
|
re="${entry%%|*}"
|
|
reason="${entry#*|}"
|
|
if grep -qE "$re" <<<"$name"; then
|
|
matched="$reason"
|
|
break
|
|
fi
|
|
done
|
|
if [ -n "$matched" ]; then
|
|
echo " DECLARED SKIP $name"
|
|
echo " -> $matched"
|
|
else
|
|
echo " UNDECLARED SKIP $name" >&2
|
|
bad=1
|
|
fi
|
|
done < <(sed -n 's/^[[:space:]]*--- SKIP: \([^[:space:]]*\).*/\1/p' "$LOG" | sort -u)
|
|
|
|
if [ "$bad" -ne 0 ]; then
|
|
echo " FAILED [$label]: the test(s) above called t.Skip and are not declared in" >&2
|
|
echo " SKIP_DECLARED at the top of this script. A skipped test is a test that" >&2
|
|
echo " DID NOT RUN, and the package's 'ok' line says nothing about it. Either" >&2
|
|
echo " make it run here, or declare it by name with the reason it cannot —" >&2
|
|
echo " the reason is printed on every run, so it has to hold up." >&2
|
|
return 1
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
run_suite() { # $1=label $2=extra go-test flags (may be empty) $3..=packages
|
|
local label="$1" extra="$2"
|
|
shift 2
|
|
local pkgs=("$@") rc=0 expect missing=0 pkg timed_out=0 t0=$SECONDS
|
|
|
|
expect="$(go list -tags "$SHATER_ROUTER_TAGS" \
|
|
-f '{{if or .TestGoFiles .XTestGoFiles}}{{.ImportPath}}{{end}}' \
|
|
"${pkgs[@]}" | grep -v '^$' || true)"
|
|
if [ -z "$expect" ]; then
|
|
echo " FAILED [$label]: go list reports no package with tests here — the gate" >&2
|
|
echo " would have run nothing and passed." >&2
|
|
FAILED=1
|
|
return
|
|
fi
|
|
echo " packages with tests: $(wc -l <<<"$expect" | tr -d ' ')"
|
|
|
|
set +e
|
|
# -v is NOT optional: it is the only way a t.Skip becomes visible at all (see
|
|
# check_skips). It costs no test TIME — measured 2026-07-26 over the fork's
|
|
# trees, warm cache, three alternating runs each: 38/25/24 s plain against
|
|
# 38/24/24 s with -v. What it costs is OUTPUT: 5 KB -> 257 KB, which is why the
|
|
# printing below is filtered rather than the flag dropped.
|
|
# shellcheck disable=SC2086 # $extra is a deliberate word-split flag list
|
|
go test -count=1 -v -timeout "$GOTIMEOUT" $extra \
|
|
-tags "$SHATER_ROUTER_TAGS" -ldflags "$SHATER_ROUTER_LDFLAGS" \
|
|
"${pkgs[@]}" >"$LOG" 2>&1
|
|
rc=$?
|
|
set -e
|
|
if [ "$rc" -ne 0 ] && grep -q '^panic: test timed out after ' "$LOG"; then
|
|
timed_out=1
|
|
fi
|
|
if [ "$timed_out" -eq 1 ]; then
|
|
# For a deadline the -v transcript ahead of the panic is a quarter of a
|
|
# megabyte of PASS lines that say nothing about a block; the goroutine dump
|
|
# the panic prints says everything. Print from the panic on.
|
|
sed -n '/^panic: test timed out after /,$p' "$LOG" | sed 's/^/ /'
|
|
elif [ "$rc" -ne 0 ]; then
|
|
# A failure needs the whole story, t.Logf output and all.
|
|
sed 's/^/ /' "$LOG"
|
|
else
|
|
# A green run gets what the non-verbose gate always printed — one line per
|
|
# package — plus every skip verdict. The rest of -v's output is a
|
|
# `=== RUN`/`--- PASS` pair per test (250 KB a suite); printing it would bury
|
|
# the handful of lines anyone reads.
|
|
grep -E '^(ok|FAIL|\?)[[:space:]]|^[[:space:]]*--- SKIP: ' "$LOG" | sed 's/^/ /' || true
|
|
fi
|
|
|
|
if [ "$rc" -ne 0 ]; then
|
|
# TIMED OUT and FAILED both exit non-zero, and until 2026-07-27 this gate
|
|
# printed the same "FAILED [race]: go test exited 1" for both. They are not
|
|
# the same event and they call for OPPOSITE actions: a failed assertion says
|
|
# the product is wrong, a deadline says nothing at all about the product
|
|
# until you know whether the tests were blocked or merely slow. Reading the
|
|
# first as the second is how a real hang gets "fixed" with a bigger number.
|
|
if [ "$timed_out" -eq 1 ]; then
|
|
echo " TIMED OUT [$label] after ${GOTIMEOUT} — NOT a failing assertion. No test said" >&2
|
|
echo " the product is wrong; go test's per-package deadline fired." >&2
|
|
echo " Still running when it did:" >&2
|
|
sed -n '/^[[:space:]]*running tests:/,/^$/p' "$LOG" \
|
|
| sed -n 's/^[[:space:]]*\(Test[^[:space:]]*.*\)$/ \1/p' >&2
|
|
echo " Two causes, opposite fixes:" >&2
|
|
echo " BLOCKED — deadlock, a channel nobody closes, a child process that" >&2
|
|
echo " never exits. The goroutine dump printed above names the line each" >&2
|
|
echo " of those tests is parked on. Fix the block; do not touch GOTIMEOUT." >&2
|
|
echo " SLOW — the suite outgrew the budget. MEASURE it before deciding:" >&2
|
|
echo " go test -race -v -timeout 30m ./<pkg>/ 2>&1 | grep -- '--- PASS'" >&2
|
|
echo " sorted by the per-test seconds. If the numbers come out as near-exact" >&2
|
|
echo " whole seconds, they are counting SLEEPS, not work — see the GOTIMEOUT" >&2
|
|
echo " comment at the top of this file, which is the last time that happened." >&2
|
|
echo " Raise the deadline only with the new measurement written beside it." >&2
|
|
else
|
|
echo " FAILED [$label]: go test exited $rc" >&2
|
|
fi
|
|
FAILED=1
|
|
# Name the skips anyway. A suite that failed somewhere else must not become
|
|
# a hiding place for a test that did not run — that is the same sin one
|
|
# level down, and while a red tree is being fixed is exactly when a skip
|
|
# gets added "temporarily". The verdict is already FAILED, so this only
|
|
# reports. Guarded on tests having actually run: a BUILD failure produces no
|
|
# verdicts to read, and check_skips' own control would then fire and bury
|
|
# the compiler error under a complaint about -v.
|
|
if grep -q '^=== RUN ' "$LOG"; then
|
|
check_skips "$label" || true
|
|
fi
|
|
return
|
|
fi
|
|
|
|
while read -r pkg; do
|
|
[ -n "$pkg" ] || continue
|
|
grep -qE "^ok[[:space:]]+$pkg([[:space:]]|\$)" "$LOG" || {
|
|
echo " DID NOT RUN [$label]: $pkg" >&2
|
|
missing=1
|
|
}
|
|
done <<<"$expect"
|
|
if [ "$missing" -ne 0 ]; then
|
|
echo " FAILED [$label]: package(s) above have test files but produced no 'ok'" >&2
|
|
echo " line. Build-constraint or file-name drift emptied them." >&2
|
|
FAILED=1
|
|
return
|
|
fi
|
|
if ! check_skips "$label"; then
|
|
FAILED=1
|
|
return
|
|
fi
|
|
# The elapsed seconds are printed on purpose: nothing here recorded a suite's
|
|
# total before, so the only thing that ever noticed [4/7] growing towards ten
|
|
# minutes was the deadline going off. One number per suite makes a doubling
|
|
# visible on the run that causes it.
|
|
echo " OK [$label] in $((SECONDS - t0))s"
|
|
}
|
|
|
|
# --- [2/7] the fork's trees, shipped tags, linux -----------------------------
|
|
echo "== [2/7] go test — the fork's trees (shipped tags, linux) =="
|
|
run_suite main "" "${ROOTS[@]}"
|
|
echo
|
|
|
|
# --- [3/7] common/, minus the tests that need CAP_NET_ADMIN ------------------
|
|
echo "== [3/7] go test — common/ (minus the CAP_NET_ADMIN integration tests) =="
|
|
run_suite common "-skip $SKIP_COMMON" "${ROOTS_COMMON[@]}"
|
|
echo
|
|
|
|
# --- [4/7] -race over the same trees -----------------------------------------
|
|
# Everything, not a subset: shater/netplane is the single most concurrency-critical
|
|
# package we own (the nft data plane), so once it is in, adding the rest is cheap.
|
|
# common/ is left out — it is upstream code exercised by upstream CI.
|
|
#
|
|
# The "~110 s for netplane" that stood here was wrong by six times: measured
|
|
# 2026-07-27 it was 663.8 s, which is what fired the deadline. See GOTIMEOUT at
|
|
# the top for what those seconds actually were and why they are now 10.6 s.
|
|
if [ "$RACE" -eq 1 ]; then
|
|
echo "== [4/7] go test -race — the fork's trees =="
|
|
echo " nothing is skipped under -race"
|
|
|
|
run_suite race "-race -skip $RACE_SKIP" "${ROOTS[@]}"
|
|
else
|
|
echo "== [4/7] -race pass skipped (--no-race) =="
|
|
fi
|
|
echo
|
|
|
|
# --- [5/7] the privileged tests may not skip in silence ----------------------
|
|
# THE HOLE THIS CLOSES. Some tests can only prove what they claim against a real
|
|
# kernel: shater/generate's TestIntegrationL3TunInboundStarts opens /dev/net/tun
|
|
# and stands a real engine on it, TestIntegrationL3EgressICMPIsAFlow binds a real
|
|
# socket to a real device. Both guard themselves with t.Skip when root or the
|
|
# device is missing — the honest thing for a test to do, and completely INVISIBLE
|
|
# above: `go test` prints `ok <pkg>` whether they ran or skipped, so [2/7]'s
|
|
# per-package `ok` check is satisfied either way and the gate closes by claiming
|
|
# it "passes every test we own". That is precisely the failure this whole script
|
|
# was written for (115 of 116 test files never running while CI stayed green),
|
|
# one level down and harder to see.
|
|
#
|
|
# The list is DISCOVERED, not hand-kept — `go test -list` over the same ROOTS —
|
|
# so a privileged test written next month joins this check on the day it is
|
|
# named, with no edit here. It keys on the ^TestIntegration prefix, already this
|
|
# fork's marker for "needs capabilities the ordinary gate lacks" (SKIP_COMMON
|
|
# above excludes common/tlsspoof's TestIntegration* for exactly that reason).
|
|
# Name a privileged test anything else and it is invisible again — so don't.
|
|
#
|
|
# Verdicts, per test, by name:
|
|
# RAN — it executed here; printed so that is visible rather than assumed.
|
|
# FAILED — fatal, like any other failure.
|
|
# MISSING — `go test -list` named it and the run produced no verdict for it:
|
|
# fatal. A test that vanished between listing and running is the
|
|
# same class of hole as one hidden by a build tag.
|
|
# SKIPPED while this environment HAS root and /dev/net/tun — fatal. The
|
|
# capability guard cannot be what skipped it, so something else did
|
|
# and only the test knows what.
|
|
# SKIPPED because the environment genuinely cannot run it — reported loudly,
|
|
# by name, and it REPLACES the closing banner, so the last line of
|
|
# the gate can never claim coverage it does not have. Deliberately
|
|
# not fatal by default: the act_runner is an LXC guest whose kernel
|
|
# has no tun module at all (checked 2026-07-26 on 10.10.10.211 —
|
|
# `modprobe tun` answers "Module tun not found", /dev/net does not
|
|
# exist, and act_runner runs job containers with privileged:false
|
|
# and no container.options), so the device cannot be handed down
|
|
# without reconfiguring the Proxmox host. Making it fatal would
|
|
# paint CI permanently red and teach everyone to ignore the gate.
|
|
# SHATER_REQUIRE_PRIVILEGED=1 makes it fatal for the runs that can.
|
|
echo "== [5/7] the privileged tests (^TestIntegration) produced a verdict by name =="
|
|
PRIV_RE='^TestIntegration'
|
|
PRIV_UNVERIFIED=""
|
|
# -ldflags is NOT optional on the discovery call either: `go test -list` LINKS
|
|
# each test binary before it can enumerate its tests, and without
|
|
# -checklinkname=0 every package that pulls common/badtls fails to link. The
|
|
# first cut of this step omitted it, swallowed the error with `2>/dev/null ||
|
|
# true`, and reported "none declared" — a check against silent skipping that was
|
|
# itself silently skipping. Hence also: the exit status is inspected, and an
|
|
# empty list is only ever reported after a SUCCESSFUL enumeration.
|
|
set +e
|
|
priv_expect_raw="$(go test -list "$PRIV_RE" \
|
|
-tags "$SHATER_ROUTER_TAGS" -ldflags "$SHATER_ROUTER_LDFLAGS" "${ROOTS[@]}" 2>&1)"
|
|
priv_list_rc=$?
|
|
set -e
|
|
priv_expect="$(grep -E "$PRIV_RE" <<<"$priv_expect_raw" | sort -u || true)"
|
|
if [ "$priv_list_rc" -ne 0 ]; then
|
|
echo " FAILED [privileged]: could not enumerate the privileged tests (go test -list exited $priv_list_rc)." >&2
|
|
echo " An unreadable list is NOT an empty list — this check refuses to" >&2
|
|
echo " report 'nothing to verify' on the strength of a failed command." >&2
|
|
sed 's/^/ /' <<<"$priv_expect_raw" | grep -vE '^\s+(ok|\?)\s' >&2 || true
|
|
FAILED=1
|
|
elif [ -z "$priv_expect" ]; then
|
|
echo " none declared under the fork's trees — nothing to verify"
|
|
else
|
|
priv_capable=0
|
|
if [ "$(id -u)" = "0" ] && [ -e /dev/net/tun ]; then
|
|
priv_capable=1
|
|
fi
|
|
echo " declared: $(wc -l <<<"$priv_expect" | tr -d ' ')"
|
|
echo " this environment: uid=$(id -u), /dev/net/tun $([ -e /dev/net/tun ] && echo present || echo MISSING) => can run them: $([ "$priv_capable" -eq 1 ] && echo yes || echo NO)"
|
|
set +e
|
|
go test -count=1 -v -run "$PRIV_RE" \
|
|
-tags "$SHATER_ROUTER_TAGS" -ldflags "$SHATER_ROUTER_LDFLAGS" \
|
|
"${ROOTS[@]}" >"$LOG" 2>&1
|
|
priv_rc=$?
|
|
set -e
|
|
# The verdict lines plus whatever reason the test printed just before them,
|
|
# so a skip is readable here and not just counted.
|
|
grep -E '^(--- (PASS|SKIP|FAIL): |[[:space:]]+[^[:space:]]+\.go:[0-9]+: )' "$LOG" \
|
|
| sed 's/^/ | /' || true
|
|
priv_bad=0
|
|
while read -r name; do
|
|
[ -n "$name" ] || continue
|
|
if grep -qE "^--- PASS: ${name}([[:space:]]|\$)" "$LOG"; then
|
|
echo " RAN $name"
|
|
elif grep -qE "^--- FAIL: ${name}([[:space:]]|\$)" "$LOG"; then
|
|
echo " FAILED $name" >&2
|
|
priv_bad=1
|
|
elif grep -qE "^--- SKIP: ${name}([[:space:]]|\$)" "$LOG"; then
|
|
if [ "$priv_capable" -eq 1 ]; then
|
|
echo " SKIPPED $name — but this environment HAS root and /dev/net/tun, so the capability guard is NOT what skipped it" >&2
|
|
priv_bad=1
|
|
else
|
|
echo " DID NOT RUN $name — skipped: no root and/or no /dev/net/tun here"
|
|
PRIV_UNVERIFIED="$PRIV_UNVERIFIED $name"
|
|
fi
|
|
else
|
|
echo " MISSING $name — go test -list named it, the run produced no verdict for it" >&2
|
|
priv_bad=1
|
|
fi
|
|
done <<<"$priv_expect"
|
|
# The runner's own exit status is reported WHATEVER the per-name verdicts say.
|
|
# It used to be reported only when every name was accounted for, on the
|
|
# reasoning that a named failure already explains the status — but the case
|
|
# that actually happens is the opposite one: the run dies at package level, the
|
|
# loop above prints MISSING for every name because none of them produced a
|
|
# verdict, and the one line naming the cause was the one suppressed. A reader
|
|
# then goes looking for three vanished tests instead of at the build error.
|
|
# Same verdict either way (priv_bad, hence FAILED, is set in both branches) —
|
|
# what changes is whether the log says why.
|
|
if [ "$priv_rc" -ne 0 ]; then
|
|
if [ "$priv_bad" -eq 0 ]; then
|
|
echo " FAILED [privileged]: go test exited $priv_rc with every named test accounted for —" >&2
|
|
echo " a build or package-level failure, see the log above." >&2
|
|
else
|
|
echo " FAILED [privileged]: go test exited $priv_rc as well — the verdicts above are" >&2
|
|
echo " the symptom. A run that dies at package level names no test, so" >&2
|
|
echo " MISSING lines are what that looks like from here; the cause is in" >&2
|
|
echo " the log above, not in the tests they name." >&2
|
|
fi
|
|
priv_bad=1
|
|
fi
|
|
if [ "$priv_bad" -ne 0 ]; then
|
|
FAILED=1
|
|
fi
|
|
fi
|
|
echo
|
|
|
|
# --- [6/7] every non-Go test file is claimed by a runner ---------------------
|
|
# See NONGO_TEST_RUNNERS above for why this exists. The instrument is `git
|
|
# ls-files`, not a filesystem walk: a test file that is not committed is not
|
|
# anybody's coverage, and a walk would also drag node_modules in.
|
|
#
|
|
# The candidate pattern is a CLOSED positive list of the shapes a test file takes
|
|
# in this repo and the ones it plausibly will (js/ts/jsx/tsx, python, bats, shell,
|
|
# ucode). It is deliberately wider than what exists today: the whole point is to
|
|
# catch the file somebody adds next month in a language no step here knows about.
|
|
echo "== [6/7] every non-Go test file is claimed by a runner =="
|
|
NONGO_TEST_RE='(\.(test|spec)\.(js|mjs|cjs|jsx|ts|tsx)|(^|/)test_[^/]*\.py|_test\.py|\.bats|_test\.sh|_test\.uc)$'
|
|
set +e
|
|
tracked="$(git ls-files 2>&1)"
|
|
tracked_rc=$?
|
|
set -e
|
|
if [ "$tracked_rc" -ne 0 ]; then
|
|
echo " FAILED [claimed]: could not enumerate the tree (git ls-files exited $tracked_rc)." >&2
|
|
echo " An unreadable list is NOT an empty list — same rule as [5/7]." >&2
|
|
sed 's/^/ /' <<<"$tracked" >&2
|
|
FAILED=1
|
|
else
|
|
nongo="$(grep -E "$NONGO_TEST_RE" <<<"$tracked" | sort || true)"
|
|
if [ -z "$nongo" ]; then
|
|
echo " FAILED [claimed]: not one non-Go test file found in a tree that has several." >&2
|
|
echo " The pattern stopped matching; this check would pass having looked" >&2
|
|
echo " at nothing." >&2
|
|
FAILED=1
|
|
else
|
|
echo " non-Go test files: $(wc -l <<<"$nongo" | tr -d ' ')"
|
|
unclaimed=0
|
|
while read -r f; do
|
|
[ -n "$f" ] || continue
|
|
owner=""
|
|
for entry in "${NONGO_TEST_RUNNERS[@]}"; do
|
|
if grep -qE "${entry%%|*}" <<<"$f"; then
|
|
owner="${entry#*|}"
|
|
break
|
|
fi
|
|
done
|
|
if [ -n "$owner" ]; then
|
|
echo " claimed $f"
|
|
echo " -> $owner"
|
|
else
|
|
echo " UNCLAIMED $f — no runner in this gate executes it" >&2
|
|
unclaimed=1
|
|
fi
|
|
done <<<"$nongo"
|
|
if [ "$unclaimed" -ne 0 ]; then
|
|
echo " FAILED [claimed]: the file(s) above are test files that NOTHING runs." >&2
|
|
echo " That is a test which cannot fail — the most expensive kind, because" >&2
|
|
echo " it reads as coverage. Either wire a runner (JS_SUITES below, or" >&2
|
|
echo " scripts/run-panel-tests.sh) and declare it in NONGO_TEST_RUNNERS, or" >&2
|
|
echo " delete the file. Declaring it without wiring one is not an option:" >&2
|
|
echo " the runner named there is the one [7/7] reports a verdict for." >&2
|
|
FAILED=1
|
|
fi
|
|
fi
|
|
fi
|
|
echo
|
|
|
|
# --- [7/7] the non-Go suites this gate owns, with a verdict by name ----------
|
|
# The work is in js_step() at the top of this file; see there for what `node
|
|
# <file>` proves and what it cannot.
|
|
#
|
|
# NODE MAY BE ABSENT, and that is handled the way [5/7] handles a missing
|
|
# /dev/net/tun: the files are named, the run says out loud that they DID NOT RUN,
|
|
# and that notice REPLACES the closing banner so this script can never end by
|
|
# claiming coverage it does not have. Not fatal by default, because the
|
|
# golang:1.26 image the non-linux re-exec uses has no node. CI does: both
|
|
# .gitea/workflows/test.yml and release.yml run actions/setup-node@v4 (node 24)
|
|
# and scripts/run-panel-tests.sh BEFORE this script, in the same job, so on the
|
|
# release path node is on PATH here and these really execute.
|
|
#
|
|
# SHATER_JS_ALREADY_RAN=1 means the re-exec that started this container ran them
|
|
# on the host first and will fold their verdict into its own exit status — so
|
|
# re-running them here would only be slower and, without node, would print a
|
|
# "did not run" that is not true of this gate as a whole.
|
|
echo "== [7/7] the non-Go suites (node) produced a verdict by name =="
|
|
if [ "${SHATER_JS_ALREADY_RAN:-0}" = "1" ]; then
|
|
echo " already run on the host before the re-exec into this container (see above);"
|
|
echo " that run's verdict is folded into the exit status of the script that started it."
|
|
else
|
|
js_step "here" || FAILED=1
|
|
fi
|
|
echo
|
|
|
|
if [ "$FAILED" -ne 0 ]; then
|
|
echo "== TEST GATE FAILED — nothing may be published from this run. ==" >&2
|
|
exit 1
|
|
fi
|
|
if [ -n "$PRIV_UNVERIFIED" ] || [ -n "$JS_UNVERIFIED" ]; then
|
|
echo "== !! PASSED, BUT NOT FULLY VERIFIED !! =================================="
|
|
echo " Every test that COULD run here passed. These did not run at all:"
|
|
for t in $PRIV_UNVERIFIED $JS_UNVERIFIED; do
|
|
echo " - $t"
|
|
done
|
|
echo
|
|
fi
|
|
if [ -n "$JS_UNVERIFIED" ]; then
|
|
echo " The file(s) above with a path are non-Go suites and this environment has"
|
|
echo " no \`node\`. The golang image the non-linux re-exec uses does not ship one;"
|
|
echo " CI does (actions/setup-node@v4, node 24, in the same job before this"
|
|
echo " script), so on the release path they DO run. To run them here:"
|
|
echo " node openwrt/luci-app-shater/tests/status-readout.test.js"
|
|
echo
|
|
fi
|
|
if [ -n "$PRIV_UNVERIFIED" ]; then
|
|
echo " The named tests above need root + CAP_NET_ADMIN + /dev/net/tun, which"
|
|
echo " this environment does not have. Nothing about the kernel paths they"
|
|
echo " cover was verified by this run. To actually run them, from a host whose"
|
|
echo " docker can:"
|
|
echo " scripts/run-tests.sh # the re-exec hands the container both"
|
|
echo " or directly:"
|
|
echo " docker run --rm --cap-add NET_ADMIN --device /dev/net/tun \\"
|
|
echo " -v \"\$PWD\":/src -w /src golang:1.26 bash scripts/run-tests.sh"
|
|
echo " or on the OpenWrt VM. SHATER_REQUIRE_PRIVILEGED=1 makes this a hard"
|
|
echo " failure instead of this notice."
|
|
fi
|
|
if [ -n "$PRIV_UNVERIFIED" ] || [ -n "$JS_UNVERIFIED" ]; then
|
|
echo "=========================================================================="
|
|
if [ -n "$PRIV_UNVERIFIED" ] && [ "${SHATER_REQUIRE_PRIVILEGED:-0}" = "1" ]; then
|
|
echo "== TEST GATE FAILED: SHATER_REQUIRE_PRIVILEGED=1 and the tests above did not run. ==" >&2
|
|
exit 1
|
|
fi
|
|
exit 0
|
|
fi
|
|
echo "== OK: the shipped tag set, on linux, passes every test we own. =="
|