fix(health): a chain blocked at a hop is dead on the board, unknown on the card
The short-circuit left the chain's exit tag untested, because the exit is itself a hop and every hop behind the break was rewritten that way. A stand run caught it — the test lives in a file that does not compile on the dev host, so nothing local could have. That is not neutral silence. selectExcluding ranks untested ABOVE dead and says so in its own comment: with no fresh-alive member, an untested one is a better bet than a known-dead one. Leaving a provably broken path untested is therefore a positive preference for it over a path we merely know is dead. The two readings answer different questions and now differ on purpose. Is this hop's own node alive — unknown behind a break, so the card keeps untested and blocked_by. Can this chain carry traffic — known, no, because the hop in front of it was probed and did not answer. The board carries that second answer, which is the one selection, the freshness gate and the manual test all read. The exit verdict is derived, not dialled: it records the consequence of a probe that did happen one hop earlier, and it is re-derived every pass, so the moment the blocker answers the walk reaches the exit again and the next verdict there is a real measurement. Also keeps a routed group warm. Its checker used to stop on the idle timeout and nothing filled in behind it, so a rule that fires rarely would show untested while being in force and pay a cold probe on the first real request. The gate that adds this work answers false when it does not know — the mirror of the one that withholds work, so plain sing-box keeps the lifecycle it always had. And the tls-spoof suite now skips without tcpdump instead of failing sixteen times: a missing tool is not measured, not broken. The same distinction this commit is about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -25,6 +25,21 @@ func requireRoot(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// requireTCPDump skips when tcpdump is not installed.
|
||||
//
|
||||
// The same honesty this package's callers demand of a health reading: a missing
|
||||
// INSTRUMENT is "not checked", never "broken". Without it every test in this
|
||||
// file fails on `cmd.Start()` — sixteen red results that say nothing about the
|
||||
// code and hide any real failure among them — on a machine where the only thing
|
||||
// wrong is that a capture tool is absent. requireRoot has always drawn that line
|
||||
// for privileges; this draws it for the tool.
|
||||
func requireTCPDump(t *testing.T) {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("tcpdump"); err != nil {
|
||||
t.Skip("integration test requires tcpdump on PATH; install it to run this suite")
|
||||
}
|
||||
}
|
||||
|
||||
func tcpdumpObserver(t *testing.T, iface string, port uint16, needle string, do func(), wait time.Duration) bool {
|
||||
t.Helper()
|
||||
return tcpdumpObserverMulti(t, iface, port, []string{needle}, do, wait)[needle]
|
||||
@@ -36,6 +51,9 @@ func tcpdumpObserver(t *testing.T, iface string, port uint16, needle string, do
|
||||
// the wire.
|
||||
func tcpdumpObserverMulti(t *testing.T, iface string, port uint16, needles []string, do func(), wait time.Duration) map[string]bool {
|
||||
t.Helper()
|
||||
// Every capture in this file funnels through here, so one guard covers the
|
||||
// whole suite and no future test can forget it.
|
||||
requireTCPDump(t)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), wait)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, "tcpdump", "-i", iface, "-n", "-A", "-l",
|
||||
|
||||
@@ -39,4 +39,23 @@ type ProbeGate interface {
|
||||
// measure its way out of that. A nil ProbeGate means "no gate" and every
|
||||
// probe proceeds.
|
||||
ProbeAllowed(tag string) bool
|
||||
|
||||
// ProbeWhenIdle reports whether the outbound tagged tag must keep measuring
|
||||
// even when no traffic is passing through it.
|
||||
//
|
||||
// A urltest group normally probes only while it is in use: Touch arms the
|
||||
// ticker on a dial, and the idle timeout stops it again. That is right for a
|
||||
// group whose readings matter only while somebody is dialling it, and wrong
|
||||
// for one the routing config REACHES: a rule that matches rarely — a narrow
|
||||
// domain list, say — is in force the whole time, so the health of its target
|
||||
// is a live question the whole time. Letting it go quiet means the panel
|
||||
// reports "untested" about a rule that is armed, and the first real request
|
||||
// pays a cold probe instead of picking an already-known-good member.
|
||||
//
|
||||
// Unlike ProbeAllowed, the safe answer here is FALSE when nothing is known.
|
||||
// This one ADDS work, and a gate that claimed it on missing information would
|
||||
// keep every group in the process probing forever — not a default anybody
|
||||
// asked for. Absent gate, unknown tag, nothing configured yet: false, and the
|
||||
// idle timeout behaves exactly as it always has.
|
||||
ProbeWhenIdle(tag string) bool
|
||||
}
|
||||
|
||||
@@ -397,6 +397,31 @@ func (g *URLTestGroup) scheduledCheck() {
|
||||
g.CheckOutbounds(false)
|
||||
}
|
||||
|
||||
// keepWarm reports whether this group must keep measuring with no traffic
|
||||
// flowing through it. lx: health board §5.C — see urltest.ProbeGate.ProbeWhenIdle.
|
||||
//
|
||||
// The default is NO, in every direction: no gate, no tag, or a group whose
|
||||
// self-check is stood down anyway. Only a gate that positively says "the routing
|
||||
// config reaches this group" turns the idle timeout off, so plain sing-box and
|
||||
// every hand-built group keep the lifecycle they have always had.
|
||||
func (g *URLTestGroup) keepWarm() bool {
|
||||
if g.selfCheckDisabled || g.probeGate == nil || g.tag == "" {
|
||||
return false
|
||||
}
|
||||
return g.probeGate.ProbeWhenIdle(g.tag)
|
||||
}
|
||||
|
||||
// startTickerLocked arms the group's own probing ticker. g.access MUST be held
|
||||
// and g.ticker MUST be nil. Extracted so PostStart and Touch arm it identically
|
||||
// — two ways in, one construction, no chance of one of them forgetting the pause
|
||||
// registration.
|
||||
func (g *URLTestGroup) startTickerLocked() {
|
||||
ticker := time.NewTicker(g.interval)
|
||||
g.ticker = ticker
|
||||
g.pauseCallback = pause.RegisterTicker(g.pause, ticker, g.interval, nil)
|
||||
go g.loopCheck(ticker, g.close)
|
||||
}
|
||||
|
||||
func NewURLTestGroup(ctx context.Context, outboundManager adapter.OutboundManager, logger log.Logger, outbounds []adapter.Outbound, link string, interval time.Duration, tolerance uint16, idleTimeout time.Duration, interruptExternalConnections bool) (*URLTestGroup, error) {
|
||||
if interval == 0 {
|
||||
interval = C.DefaultURLTestInterval
|
||||
@@ -454,6 +479,21 @@ func (g *URLTestGroup) PostStart() {
|
||||
// sweep would measure that hop once per member and file the result against
|
||||
// this one. selfCheckAllowed keeps both refusals in one place.
|
||||
go g.scheduledCheck()
|
||||
// A group the routing config REACHES keeps measuring whether or not anybody
|
||||
// dials it, so its ticker is armed here instead of waiting for a Touch that
|
||||
// may never come. Without this, a used group with no traffic gets this one
|
||||
// warm-up sweep and then nothing: its members age past the verdict TTL and
|
||||
// the panel reports "untested" about a rule that is in force, while the first
|
||||
// real request pays a cold probe. Nothing else would fill the gap — the
|
||||
// observatory stands off a urltest group's members entirely (probeplan.go
|
||||
// SelfChecked), which is the whole point of one dialler per target.
|
||||
//
|
||||
// lastActive was stored a moment ago, so loopCheck's opening "idle longer
|
||||
// than the interval" check does not fire and this cannot double up with the
|
||||
// sweep above.
|
||||
if g.keepWarm() && g.ticker == nil {
|
||||
g.startTickerLocked()
|
||||
}
|
||||
}
|
||||
|
||||
func (g *URLTestGroup) Touch() {
|
||||
@@ -483,10 +523,7 @@ func (g *URLTestGroup) Touch() {
|
||||
g.lastActive.Store(time.Now())
|
||||
return
|
||||
}
|
||||
ticker := time.NewTicker(g.interval)
|
||||
g.ticker = ticker
|
||||
g.pauseCallback = pause.RegisterTicker(g.pause, ticker, g.interval, nil)
|
||||
go g.loopCheck(ticker, g.close)
|
||||
g.startTickerLocked()
|
||||
}
|
||||
|
||||
func (g *URLTestGroup) Close() error {
|
||||
@@ -528,7 +565,13 @@ func (g *URLTestGroup) loopCheck(ticker *time.Ticker, closeChan <-chan struct{})
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
if time.Since(g.lastActive.Load()) > g.idleTimeout {
|
||||
// The idle timeout retires the ticker of a group nobody is dialling —
|
||||
// unless the routing config reaches it, in which case its health is a
|
||||
// live question whether or not traffic is flowing and the ticker must
|
||||
// outlive the silence. Asked here rather than remembered from PostStart
|
||||
// so it tracks the running config, and asked OUTSIDE g.access because the
|
||||
// answer comes from the engine, which has locks of its own.
|
||||
if !g.keepWarm() && time.Since(g.lastActive.Load()) > g.idleTimeout {
|
||||
g.access.Lock()
|
||||
if g.ticker == ticker {
|
||||
g.ticker.Stop()
|
||||
|
||||
@@ -128,6 +128,7 @@ func TestSelfCheckOptionPlumbing(t *testing.T) {
|
||||
type fakeGate struct {
|
||||
mu sync.Mutex
|
||||
blocked map[string]bool
|
||||
warm map[string]bool
|
||||
asked int
|
||||
}
|
||||
|
||||
@@ -138,6 +139,12 @@ func (g *fakeGate) ProbeAllowed(tag string) bool {
|
||||
return !g.blocked[tag]
|
||||
}
|
||||
|
||||
func (g *fakeGate) ProbeWhenIdle(tag string) bool {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
return g.warm[tag]
|
||||
}
|
||||
|
||||
func (g *fakeGate) set(tag string, blocked bool) {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
|
||||
@@ -214,9 +214,16 @@ func (e *Engine) applyLocked(opts option.Options) (bool, error) {
|
||||
// change that flips a group used<->unused is then a real change that
|
||||
// triggers a swap, and an unchanged config hashes identically on every
|
||||
// reconcile because the stand-down is deterministic.
|
||||
if stood := standDownUnusedSelfCheck(opts); stood > 0 && e.log != nil {
|
||||
stood, used := standDownUnusedSelfCheck(opts)
|
||||
if stood > 0 && e.log != nil {
|
||||
e.log.Info("apply: stood down self-check on ", stood, " unused urltest group(s); the observatory is their only prober")
|
||||
}
|
||||
// Publish the same walk's used-set for ProbeWhenIdle BEFORE the new box is
|
||||
// built, because the question is asked during that box's PostStart: a used
|
||||
// group arms its probing ticker there rather than waiting for traffic. The
|
||||
// observatory's own copy is published later, after a SUCCESSFUL swap, which
|
||||
// is too late to be the answer.
|
||||
e.publishKeepWarm(used)
|
||||
|
||||
newHash, err := e.hashOptions(opts)
|
||||
if err != nil {
|
||||
|
||||
@@ -542,6 +542,18 @@ func (e *Engine) testOneTarget(t groupTestTarget, used map[string]bool, obsEnabl
|
||||
|
||||
deadline := time.Now().Add(groupTestWaitDeadline)
|
||||
for {
|
||||
// Asked BEFORE the board read, and only for a chain. The exit of a blocked
|
||||
// chain does carry a fresh dead verdict now — the observatory derives it
|
||||
// from the broken hop (markChainExitDead) so selection cannot be tempted
|
||||
// down a path that provably does not work — but "the probe through this
|
||||
// path failed" would be the wrong sentence to show a person: no probe of
|
||||
// the exit ran, and the actionable fact is WHICH hop stopped it. Same
|
||||
// verdict, better answer.
|
||||
if idx, ok := e.blockedChainHop(t, since); ok {
|
||||
res.OK = false
|
||||
res.Error = fmt.Sprintf(groupTestErrChainBlockedFmt, idx)
|
||||
return res
|
||||
}
|
||||
if e.readFreshObservation(&res, t, since) {
|
||||
if res.OK {
|
||||
// The exit address is best-effort by design: see GroupTestResult.
|
||||
@@ -551,14 +563,6 @@ func (e *Engine) testOneTarget(t groupTestTarget, used map[string]bool, obsEnabl
|
||||
}
|
||||
return res
|
||||
}
|
||||
// Only after the board has failed to answer: a chain whose walk is blocked
|
||||
// will never get a fresh exit observation, so waiting on one is waiting for
|
||||
// something that is not coming.
|
||||
if idx, ok := e.blockedChainHop(t, since); ok {
|
||||
res.OK = false
|
||||
res.Error = fmt.Sprintf(groupTestErrChainBlockedFmt, idx)
|
||||
return res
|
||||
}
|
||||
if !time.Now().Before(deadline) {
|
||||
res.Error = groupTestErrNotReached
|
||||
return res
|
||||
|
||||
@@ -142,6 +142,20 @@ type observatoryState struct {
|
||||
used map[string]bool
|
||||
cursor int
|
||||
cycles uint64
|
||||
// keepWarm is the set of tags the routing config REACHES, and it answers
|
||||
// ProbeWhenIdle: a group in it keeps its own probing ticker running with no
|
||||
// traffic flowing.
|
||||
//
|
||||
// It is deliberately NOT the same field as used above, even though both hold
|
||||
// a used-set. used is published only after a SUCCESSFUL apply, because the
|
||||
// panel's "unused" badge must describe the config that is actually running;
|
||||
// this one is published BEFORE the new box is built, because a group asks the
|
||||
// question during its own PostStart. A failed apply can therefore leave this
|
||||
// describing a config that never started — worst case a group keeps a ticker
|
||||
// it did not need until the next apply, which nobody can see and which costs
|
||||
// one probe interval. Merging the two would force one of those two timings on
|
||||
// the other, and each is right for its own reader.
|
||||
keepWarm map[string]bool
|
||||
// hops indexes the plan's chain measurements — chain name -> hops in wire
|
||||
// order -> the dial tags measuring each hop. Rebuilt from the plan whenever
|
||||
// the plan is, and read (never mutated) by a tick to decide whether a hop's
|
||||
@@ -188,6 +202,9 @@ func (e *Engine) ConfigureObservatory(cfg ObservatoryConfig) {
|
||||
|
||||
if !cfg.Enabled {
|
||||
e.obs.plan, e.obs.used, e.obs.cursor, e.obs.hops = nil, nil, 0, nil
|
||||
// The master switch is off: nothing probes in the background, so no group
|
||||
// keeps a ticker alive on the background prober's behalf either.
|
||||
e.obs.keepWarm = nil
|
||||
e.obs.force = false
|
||||
if e.obs.stop != nil {
|
||||
close(e.obs.stop)
|
||||
@@ -236,6 +253,7 @@ func (e *Engine) StopObservatory() {
|
||||
e.obs.stop, e.obs.done, e.obs.nudge = nil, nil, nil
|
||||
e.obs.enabled = false
|
||||
e.obs.plan, e.obs.used, e.obs.cursor, e.obs.hops = nil, nil, 0, nil
|
||||
e.obs.keepWarm = nil
|
||||
e.obs.force = false
|
||||
if stop != nil {
|
||||
close(stop)
|
||||
@@ -503,7 +521,8 @@ func (e *Engine) probeChainOrdered(
|
||||
// Re-read the board here, not once up front: the hop before this one may
|
||||
// have been probed moments ago by the loop's previous iteration, and its
|
||||
// verdict is exactly what decides this one.
|
||||
if _, blocked := idx.blockedBefore(walk.chain, hop.index, e.HealthView()); blocked {
|
||||
if blocker, blocked := idx.blockedBefore(walk.chain, hop.index, e.HealthView()); blocked {
|
||||
e.markChainExitDead(walk.chain, idx, blocker)
|
||||
return // hops are ascending, so everything left is behind the same break
|
||||
}
|
||||
var wg sync.WaitGroup
|
||||
@@ -529,6 +548,69 @@ func (e *Engine) probeChainOrdered(
|
||||
}
|
||||
}
|
||||
|
||||
// markChainExitDead records a DEAD verdict for a chain's EXIT on the shared
|
||||
// health board when the ordered walk stopped short of it.
|
||||
//
|
||||
// # Two questions, two answers, and why they differ
|
||||
//
|
||||
// "Is this hop's own node alive?" and "can this chain carry traffic?" are not the
|
||||
// same question, and behind a break they have different answers:
|
||||
//
|
||||
// - the HOP's own health is genuinely UNKNOWN. Nothing dialled it, so its card
|
||||
// reads untested with BlockedBy naming the blocker (markBlockedHops), and
|
||||
// that stays exactly as it is;
|
||||
// - the CHAIN's health is KNOWN, and the answer is no. A chain is a series
|
||||
// path; hop k was probed and did not answer — a positive finding — and every
|
||||
// dial through the chain crosses hop k. The exit cannot carry traffic. That
|
||||
// is not absence of knowledge, it is a conclusion drawn from a measurement
|
||||
// somebody took.
|
||||
//
|
||||
// # Why the board must say so
|
||||
//
|
||||
// The board is not only the panel's data source; the strategies select on it, and
|
||||
// they rank UNTESTED ABOVE DEAD on purpose (protocol/group/urltest_health_lx.go
|
||||
// selectExcluding: "an untested one is a better bet than a known-dead one"). So
|
||||
// leaving a provably-broken path untested is not a neutral silence — it is an
|
||||
// invitation to route traffic down it, which is worse than the mislabelling this
|
||||
// whole change set exists to remove. Marking it keeps
|
||||
// TestChainDeadExitMarkedByObservatory's older invariant intact too: a dead chain
|
||||
// is marked dead, and a dead entry stays distinguishable from a never-measured
|
||||
// one.
|
||||
//
|
||||
// # Why this is not a fabricated verdict
|
||||
//
|
||||
// Nothing here claims a probe happened. It records the CONSEQUENCE of a probe
|
||||
// that did happen, one hop earlier, and it is re-derived from the board every
|
||||
// pass — the instant the blocking hop answers again, the walk proceeds and the
|
||||
// exit's next verdict is a real measurement of its own. The flip is logged with
|
||||
// its cause so the trail never reads as an unexplained death.
|
||||
//
|
||||
// The exit's tags are the LAST hop's measurement tags: the wrapper itself for a
|
||||
// node exit, the member copies for a group exit — in both cases exactly what a
|
||||
// reader consults to ask whether the chain works.
|
||||
func (e *Engine) markChainExitDead(chain string, idx chainPlanIndex, blocker chainPlanHop) {
|
||||
hops := idx[chain]
|
||||
if len(hops) == 0 {
|
||||
return
|
||||
}
|
||||
exit := hops[len(hops)-1]
|
||||
if exit.index <= blocker.index {
|
||||
return // the blocker IS the exit: its own probe already said so
|
||||
}
|
||||
hist := e.URLTestHistory()
|
||||
if hist == nil {
|
||||
return
|
||||
}
|
||||
ttl := e.healthTTL()
|
||||
for _, tag := range exit.tags {
|
||||
if e.log != nil && hist.Verdict(tag, ttl) != urltest.VerdictDead {
|
||||
e.log.Info("observatory: ", tag, ": -> dead (chain hop ", blocker.index,
|
||||
" was probed and did not answer, so nothing reaches the exit through it)")
|
||||
}
|
||||
hist.MarkFailed(tag)
|
||||
}
|
||||
}
|
||||
|
||||
// chainBatch is one chain's share of one batch, grouped into hops in wire order.
|
||||
type chainBatch struct {
|
||||
chain string
|
||||
@@ -704,6 +786,46 @@ func (e *Engine) ProbeAllowed(tag string) bool {
|
||||
return !blocked
|
||||
}
|
||||
|
||||
// ProbeWhenIdle implements urltest.ProbeGate: it reports whether an outbound
|
||||
// must keep measuring itself with no traffic flowing through it.
|
||||
//
|
||||
// TRUE for anything the routing configuration reaches. Such a group's health is
|
||||
// a live question at all times — a rule matching a narrow domain list is in
|
||||
// force whether or not it fired in the last half hour — and without this its own
|
||||
// ticker would retire on the idle timeout and NOTHING would replace it: the
|
||||
// observatory stands off a urltest group's members by design (ProbeJob.SelfChecked).
|
||||
// The panel would then report "untested" about an armed rule, and the first
|
||||
// request through it would pay a cold probe instead of picking a member already
|
||||
// known to work. Keeping the group's own ticker running restores exactly what
|
||||
// the observatory used to spend on those same targets, at the same interval, and
|
||||
// keeps it to ONE dialler.
|
||||
//
|
||||
// FALSE whenever nothing is known — no config applied yet, unknown tag, master
|
||||
// switch off. This method adds work rather than withholding it, so the direction
|
||||
// of "don't know" is the opposite of ProbeAllowed's: claiming keep-warm on
|
||||
// missing information would leave groups probing forever for no stated reason.
|
||||
//
|
||||
// It does NOT bypass any other refusal. A stood-down group never asks (the
|
||||
// group's own keepWarm short-circuits on selfCheckDisabled), and a kept-warm
|
||||
// group behind a dead hop still ticks WITHOUT dialling, because every tick goes
|
||||
// through scheduledCheck -> ProbeAllowed. Warm means measured on schedule, never
|
||||
// "hammering through a broken path".
|
||||
func (e *Engine) ProbeWhenIdle(tag string) bool {
|
||||
e.obsMu.Lock()
|
||||
defer e.obsMu.Unlock()
|
||||
return e.obs.keepWarm[tag]
|
||||
}
|
||||
|
||||
// publishKeepWarm installs the set ProbeWhenIdle answers from. Called from
|
||||
// applyLocked with the used-set of the config being applied, and cleared when
|
||||
// background probing is switched off — see the keepWarm field for why this is
|
||||
// published on a different schedule from the observatory's own used-set.
|
||||
func (e *Engine) publishKeepWarm(used map[string]bool) {
|
||||
e.obsMu.Lock()
|
||||
e.obs.keepWarm = used
|
||||
e.obsMu.Unlock()
|
||||
}
|
||||
|
||||
// hopBoardState rolls a hop's measurement tags into one verdict, by exactly the
|
||||
// rule chainGroupHop (grouphealth.go) uses for the same question — "can this hop
|
||||
// carry the chain": alive as soon as anything answers, dead only when something
|
||||
|
||||
@@ -16,8 +16,10 @@ import (
|
||||
"github.com/sagernet/sing-box/log"
|
||||
"github.com/sagernet/sing-box/option"
|
||||
"github.com/sagernet/sing-box/protocol/group"
|
||||
"github.com/sagernet/sing/common/json/badoption"
|
||||
M "github.com/sagernet/sing/common/metadata"
|
||||
"github.com/sagernet/sing/service"
|
||||
"github.com/sagernet/sing/service/pause"
|
||||
)
|
||||
|
||||
// obsFixture is a small applied config: one used urltest group over two nodes,
|
||||
@@ -358,7 +360,12 @@ func (p *probeRecorder) reset() {
|
||||
|
||||
// newChainProber wires an engine, the 4-hop plan and a recording stub together.
|
||||
func newChainProber(t *testing.T, dead func(string) bool) (*Engine, *probeRecorder) {
|
||||
return newChainProberOf(t, chainOptsFixture(), dead)
|
||||
t.Helper()
|
||||
e, rec := newChainProberOf(t, chainOptsFixture(), dead)
|
||||
if plan, _, _ := obsState(e); len(plan) != 7 {
|
||||
t.Fatalf("plan = %d jobs, want 7 (h1 + 2 members x 3 group hops): %+v", len(plan), plan)
|
||||
}
|
||||
return e, rec
|
||||
}
|
||||
|
||||
func newChainProberOf(t *testing.T, opts option.Options, dead func(string) bool) (*Engine, *probeRecorder) {
|
||||
@@ -375,9 +382,6 @@ func newChainProberOf(t *testing.T, opts option.Options, dead func(string) bool)
|
||||
quiesceObservatoryLoop(e)
|
||||
rec := &probeRecorder{dead: dead, hist: e.URLTestHistory()}
|
||||
e.setProbeFn(rec.probe)
|
||||
if plan, _, _ := obsState(e); len(plan) != 7 {
|
||||
t.Fatalf("plan = %d jobs, want 7 (h1 + 2 members x 3 group hops): %+v", len(plan), plan)
|
||||
}
|
||||
return e, rec
|
||||
}
|
||||
|
||||
@@ -701,8 +705,11 @@ func chainCheckers(t *testing.T, e *Engine) (map[string]*group.URLTest, *sync.Mu
|
||||
}
|
||||
}
|
||||
// The manager goes into the engine's own registry, which is exactly how a box
|
||||
// publishes it; the groups below read it back out of the same ctx.
|
||||
// publishes it; the groups below read it back out of the same ctx. The pause
|
||||
// manager likewise: a real box installs one (box.go), and a kept-warm group
|
||||
// arms a ticker at PostStart, which registers itself with it.
|
||||
service.MustRegister[adapter.OutboundManager](e.ctx, &stubOutboundManager{nodes: nodes})
|
||||
_ = pause.WithDefaultManager(e.ctx)
|
||||
|
||||
groups := map[string]*group.URLTest{}
|
||||
for hop := 2; hop <= 4; hop++ {
|
||||
@@ -961,3 +968,291 @@ func TestPlanHandsURLTestMembersToTheirGroup(t *testing.T) {
|
||||
t.Errorf("egress copy store = %v, want the copy plus its base alias", j.Store)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- keeping a used group warm ----------------------------------------------
|
||||
|
||||
// quietGroupFixture is a used urltest group that NOTHING routes traffic
|
||||
// through in practice: one rule points at it, so it is in the used-set, but the
|
||||
// test never dials it. That is the shape of a rule with a narrow match list — in
|
||||
// force the whole time, fired rarely.
|
||||
func quietGroupFixture() option.Options {
|
||||
return option.Options{
|
||||
Outbounds: []option.Outbound{
|
||||
fixNode("n1", ""),
|
||||
fixNode("n2", ""),
|
||||
fixGroup(C.TypeURLTest, "quiet", "n1", "n2"),
|
||||
},
|
||||
Route: fixRoute("quiet"),
|
||||
}
|
||||
}
|
||||
|
||||
// newQuietGroup builds the REAL protocol/group urltest group for that fixture on
|
||||
// the engine's own context, over dial-counting members, with a short interval and
|
||||
// an even shorter idle window so the idle timeout can be observed (or observed
|
||||
// not to fire) in test time rather than in half an hour.
|
||||
func newQuietGroup(t *testing.T, e *Engine, tags ...string) (map[string]*group.URLTest, *sync.Mutex, map[string]int) {
|
||||
t.Helper()
|
||||
var mu sync.Mutex
|
||||
hits := map[string]int{}
|
||||
nodes := map[string]adapter.Outbound{}
|
||||
for _, tag := range tags {
|
||||
for _, m := range []string{"n1", "n2"} {
|
||||
m = tag + "-" + m
|
||||
nodes[m] = &tallyOutbound{failingOutbound{tag: m}, &mu, hits}
|
||||
}
|
||||
}
|
||||
// n1/n2 without a prefix too: the configured fixture names them plainly.
|
||||
for _, m := range []string{"n1", "n2"} {
|
||||
nodes[m] = &tallyOutbound{failingOutbound{tag: m}, &mu, hits}
|
||||
}
|
||||
service.MustRegister[adapter.OutboundManager](e.ctx, &stubOutboundManager{nodes: nodes})
|
||||
// The ticker registers itself with the pause manager, which a real box
|
||||
// installs (box.go) and a bare engine context does not.
|
||||
_ = pause.WithDefaultManager(e.ctx)
|
||||
|
||||
out := map[string]*group.URLTest{}
|
||||
for _, tag := range tags {
|
||||
members := []string{"n1", "n2"}
|
||||
if tag != "quiet" {
|
||||
members = []string{tag + "-n1", tag + "-n2"}
|
||||
}
|
||||
ob, err := group.NewURLTest(e.ctx, nil, log.NewNOPFactory().Logger(), tag,
|
||||
option.URLTestOutboundOptions{
|
||||
Outbounds: members,
|
||||
Interval: badoption.Duration(20 * time.Millisecond),
|
||||
IdleTimeout: badoption.Duration(40 * time.Millisecond),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewURLTest(%s): %v", tag, err)
|
||||
}
|
||||
ut := ob.(*group.URLTest)
|
||||
if err := ut.Start(); err != nil {
|
||||
t.Fatalf("Start(%s): %v", tag, err)
|
||||
}
|
||||
t.Cleanup(func() { _ = ut.Close() })
|
||||
out[tag] = ut
|
||||
}
|
||||
return out, &mu, hits
|
||||
}
|
||||
|
||||
// waitForHits polls until tag has been dialled at least n times, or gives up.
|
||||
func waitForHits(mu *sync.Mutex, hits map[string]int, tag string, n int, within time.Duration) int {
|
||||
deadline := time.Now().Add(within)
|
||||
for {
|
||||
got := hitsOf(mu, hits)[tag]
|
||||
if got >= n || !time.Now().Before(deadline) {
|
||||
return got
|
||||
}
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// A group the routing config REACHES keeps measuring with no traffic at all.
|
||||
//
|
||||
// This is the coverage the one-dialler split would otherwise have cost. The
|
||||
// observatory no longer probes a urltest group's members, and the group's own
|
||||
// ticker used to retire on the idle timeout — so a rule that matches rarely
|
||||
// would have gone dark: the panel showing "untested" for a rule that is in
|
||||
// force, and the first real request paying a cold probe instead of picking a
|
||||
// member already known to work.
|
||||
//
|
||||
// Three things at once, because they only mean anything together: the readings
|
||||
// keep coming, the board never falls back to untested, and the dialling is done
|
||||
// by ONE side — the group's own checker, never the observatory.
|
||||
func TestUsedGroupKeepsMeasuringWithoutTraffic(t *testing.T) {
|
||||
e := New()
|
||||
t.Cleanup(e.StopObservatory)
|
||||
opts := quietGroupFixture()
|
||||
e.ConfigureObservatory(ObservatoryConfig{Enabled: true, Options: opts, ProbeInterval: time.Minute})
|
||||
quiesceObservatoryLoop(e)
|
||||
rec := &probeRecorder{dead: func(string) bool { return false }, hist: e.URLTestHistory()}
|
||||
e.setProbeFn(rec.probe)
|
||||
// What applyLocked publishes before the box starts: the same walk's used-set.
|
||||
_, used := standDownUnusedSelfCheck(opts)
|
||||
e.publishKeepWarm(used)
|
||||
|
||||
groups, mu, hits := newQuietGroup(t, e, "quiet")
|
||||
_ = groups["quiet"].PostStart()
|
||||
|
||||
// The idle window is 40ms and nothing ever dials this group, so an
|
||||
// unkept ticker would have retired long before the third probe.
|
||||
if got := waitForHits(mu, hits, "n1", 3, 2*time.Second); got < 3 {
|
||||
t.Fatalf("n1 was dialled %d times in 2s; a used group must keep measuring past its idle timeout", got)
|
||||
}
|
||||
|
||||
// The board carries a real verdict, not "nothing is known". These members
|
||||
// fail their dial, so dead is the honest answer — the point is that it is an
|
||||
// ANSWER, and a fresh one.
|
||||
for _, tag := range []string{"n1", "n2"} {
|
||||
state, _, age := e.HealthView().State(tag)
|
||||
if state == HealthUntested {
|
||||
t.Errorf("%s went untested while its group is reached by a rule", tag)
|
||||
}
|
||||
if age < 0 || age > 5 {
|
||||
t.Errorf("%s observation is %ds old; the schedule is not keeping it current", tag, age)
|
||||
}
|
||||
}
|
||||
|
||||
// One dialler. The observatory ran a full forced pass and dialled none of
|
||||
// this group's members: they are the group's, and they were.
|
||||
forcedTick(e)
|
||||
for _, tag := range rec.dialled() {
|
||||
if tag == "n1" || tag == "n2" {
|
||||
t.Errorf("the observatory dialled %s as well; that is the duplicate we removed", tag)
|
||||
}
|
||||
}
|
||||
if hitsOf(mu, hits)["n1"] == 0 {
|
||||
t.Error("nothing dialled n1 at all")
|
||||
}
|
||||
}
|
||||
|
||||
// The other side of the same switch: a group the config does NOT reach keeps the
|
||||
// idle behaviour it has always had. Keep-warm is a positive answer about a
|
||||
// specific tag, never a blanket "probe forever" — an unknown tag, an unconfigured
|
||||
// engine and plain sing-box all fall here.
|
||||
func TestUnreachedGroupStillIdlesOut(t *testing.T) {
|
||||
e := New()
|
||||
t.Cleanup(e.StopObservatory)
|
||||
// Nothing published: this is also what every non-shater embedder sees.
|
||||
groups, mu, hits := newQuietGroup(t, e, "quiet")
|
||||
_ = groups["quiet"].PostStart()
|
||||
|
||||
// Give it as long as the kept-warm group needed for three probes.
|
||||
got := waitForHits(mu, hits, "n1", 3, 500*time.Millisecond)
|
||||
if got != 1 {
|
||||
t.Fatalf("n1 dialled %d times, want exactly 1 (the PostStart warm-up and nothing else)", got)
|
||||
}
|
||||
if e.ProbeWhenIdle("quiet") {
|
||||
t.Error("ProbeWhenIdle said yes with nothing configured; the default must be the old behaviour")
|
||||
}
|
||||
}
|
||||
|
||||
// Warm does not mean hammering. A group kept warm because the config reaches it,
|
||||
// but sitting behind a DEAD hop, keeps its ticker and dials nothing — every tick
|
||||
// still goes through the same gate. Without this the fix for the waste would
|
||||
// have quietly reintroduced it on exactly the chains it was written for.
|
||||
func TestKeptWarmGroupBehindDeadHopStillDialsNothing(t *testing.T) {
|
||||
e, rec := newChainProberOf(t, chainOptsFixtureOf(C.TypeURLTest), func(string) bool { return false })
|
||||
_, used := standDownUnusedSelfCheck(chainOptsFixtureOf(C.TypeURLTest))
|
||||
e.publishKeepWarm(used)
|
||||
if !e.ProbeWhenIdle("chain-c-h3") {
|
||||
t.Fatal("precondition: the chain's hop wrappers must be kept warm")
|
||||
}
|
||||
|
||||
hist := e.URLTestHistory()
|
||||
hist.MarkFailed("chain-c-h2-a")
|
||||
hist.MarkFailed("chain-c-h2-b")
|
||||
|
||||
groups, mu, hits := chainCheckers(t, e)
|
||||
forcedTick(e)
|
||||
for hop := 3; hop <= 4; hop++ {
|
||||
sweep(t, groups[fmt.Sprintf("chain-c-h%d", hop)], mu, hits, 1)
|
||||
}
|
||||
|
||||
for tag, n := range hitsOf(mu, hits) {
|
||||
if n > 0 && (strings.HasPrefix(tag, "chain-c-h3") || strings.HasPrefix(tag, "chain-c-h4")) {
|
||||
t.Errorf("kept-warm wrapper dialled %s (%d times) through a dead hop 2", tag, n)
|
||||
}
|
||||
}
|
||||
for _, tag := range rec.dialled() {
|
||||
if strings.HasPrefix(tag, "chain-c-h3") || strings.HasPrefix(tag, "chain-c-h4") {
|
||||
t.Errorf("the observatory dialled %s from behind a dead hop 2", tag)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TWO QUESTIONS, TWO ANSWERS. Behind a break, "is this hop's own node alive?"
|
||||
// and "can this chain carry traffic?" stop having the same answer, and the code
|
||||
// must give each its own:
|
||||
//
|
||||
// - the HOP is genuinely unknown — nothing dialled it — so its card reads
|
||||
// untested with the blocker named;
|
||||
// - the CHAIN is known to be down. Hop k was probed and did not answer, every
|
||||
// dial through the chain crosses hop k, so the exit cannot carry traffic.
|
||||
// The shared board must SAY dead, because the strategies select on that same
|
||||
// board and rank untested ABOVE dead (selectExcluding: "an untested one is a
|
||||
// better bet than a known-dead one"). Leaving a provably broken path untested
|
||||
// is an invitation to route traffic down it.
|
||||
//
|
||||
// This is also the older invariant TestChainDeadExitMarkedByObservatory holds on
|
||||
// the stand, from before the ordered walk existed: a dead chain reads dead.
|
||||
func TestBlockedChainExitStaysDeadOnTheBoard(t *testing.T) {
|
||||
e, rec := newChainProber(t, func(dial string) bool {
|
||||
return strings.HasPrefix(dial, "chain-c-h2-")
|
||||
})
|
||||
forcedTick(e)
|
||||
|
||||
// Still no dial behind the break — the derived verdict is a conclusion, not a
|
||||
// probe, and it must not cost one.
|
||||
if got := rec.dialled(); !equalStrings(got, hopsThroughDeadH2) {
|
||||
t.Fatalf("dialled %v, want exactly %v", got, hopsThroughDeadH2)
|
||||
}
|
||||
|
||||
// The board: the exit's measurement tags read DEAD.
|
||||
hist := e.URLTestHistory()
|
||||
ttl := e.healthTTL()
|
||||
for _, tag := range []string{"chain-c-h4-a", "chain-c-h4-b"} {
|
||||
if v := hist.Verdict(tag, ttl); v != urltest.VerdictDead {
|
||||
t.Errorf("board verdict for exit tag %s = %v, want dead — selection reads this and prefers untested over dead", tag, v)
|
||||
}
|
||||
if hist.LoadURLTestHistory(tag) == nil {
|
||||
t.Errorf("exit entry %s was deleted; a dead verdict must keep it", tag)
|
||||
}
|
||||
}
|
||||
// An INTERMEDIATE hop behind the break is not the chain's verdict and is left
|
||||
// alone: nothing routes to hop 3 as such, and inventing a death for it would
|
||||
// be a claim about a node nobody measured.
|
||||
for _, tag := range []string{"chain-c-h3-a", "chain-c-h3-b"} {
|
||||
if v := hist.Verdict(tag, ttl); v != urltest.VerdictUntested {
|
||||
t.Errorf("intermediate hop tag %s = %v, want untested — only the chain's exit carries the derived verdict", tag, v)
|
||||
}
|
||||
}
|
||||
|
||||
// The card: unchanged by any of it. Hop 3 and hop 4 still read untested with
|
||||
// hop 2 named, because the question a hop row answers is the first one.
|
||||
hops := chainHopHealthOf(chainPoolFixture(), "c", e.HealthView())
|
||||
for _, h := range hops[2:] {
|
||||
if h.State != HealthUntested {
|
||||
t.Errorf("hop %d card = %q, want %q: nothing dialled it", h.Index, h.State, HealthUntested)
|
||||
}
|
||||
if h.BlockedBy == nil || h.BlockedBy.Index != 2 {
|
||||
t.Errorf("hop %d card lost its BlockedBy: %+v", h.Index, h.BlockedBy)
|
||||
}
|
||||
}
|
||||
|
||||
// And it clears itself. Once hop 2 answers, the walk reaches the exit and the
|
||||
// exit's next verdict is a measurement of its own, not a conclusion.
|
||||
e.setProbeFn((&probeRecorder{dead: func(string) bool { return false }, hist: hist}).probe)
|
||||
forcedTick(e)
|
||||
for _, tag := range []string{"chain-c-h4-a", "chain-c-h4-b"} {
|
||||
if v := hist.Verdict(tag, ttl); v != urltest.VerdictAlive {
|
||||
t.Errorf("exit tag %s = %v after hop 2 recovered, want a real alive measurement", tag, v)
|
||||
}
|
||||
}
|
||||
if h := chainHopHealthOf(chainPoolFixture(), "c", e.HealthView()); h[3].BlockedBy != nil {
|
||||
t.Errorf("the exit card still reports a block after recovery: %+v", h[3].BlockedBy)
|
||||
}
|
||||
}
|
||||
|
||||
// A NODE exit — the shape the stand's 2-hop fixture uses, where the exit wrapper
|
||||
// is a plain outbound and is itself the measurement tag. Hop 1 dies, hop 2 is
|
||||
// never dialled, and hop 2's own tag must still read dead on the board.
|
||||
func TestBlockedChainExitStaysDeadForANodeExit(t *testing.T) {
|
||||
opts := option.Options{
|
||||
Outbounds: []option.Outbound{
|
||||
fixNode("chain-two-h1", ""),
|
||||
fixNode("chain-two-h2", "chain-two-h1"),
|
||||
},
|
||||
Route: fixRoute("", "chain-two-h2"),
|
||||
}
|
||||
e, rec := newChainProberOf(t, opts, func(dial string) bool { return dial == "chain-two-h1" })
|
||||
forcedTick(e)
|
||||
|
||||
if got := rec.dialled(); !equalStrings(got, []string{"chain-two-h1"}) {
|
||||
t.Fatalf("dialled %v, want only the dead hop 1 — the exit must not be dialled through it", got)
|
||||
}
|
||||
hist := e.URLTestHistory()
|
||||
if v := hist.Verdict("chain-two-h2", e.healthTTL()); v != urltest.VerdictDead {
|
||||
t.Fatalf("exit chain-two-h2 verdict = %v, want dead (the invariant the stand's chain suite holds)", v)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -71,7 +71,13 @@ import (
|
||||
// or something foreign) is skipped rather than guessed at: it did not come from
|
||||
// our generator, and silently rebuilding somebody else's option struct is worse
|
||||
// than leaving one group's self-check up.
|
||||
func standDownUnusedSelfCheck(opts option.Options) int {
|
||||
// It also RETURNS the used-set it computed. That set is needed before the new
|
||||
// box starts — a group's PostStart asks whether it must keep measuring with no
|
||||
// traffic (Engine.ProbeWhenIdle), and the observatory does not publish its own
|
||||
// copy until after the swap has succeeded. Handing back the walk that has
|
||||
// already run here is cheaper than a second one and, more to the point, cannot
|
||||
// disagree with the flags just written from it.
|
||||
func standDownUnusedSelfCheck(opts option.Options) (int, map[string]bool) {
|
||||
_, used := BuildObservatoryPlan(opts, "")
|
||||
stood := 0
|
||||
for i := range opts.Outbounds {
|
||||
@@ -89,5 +95,5 @@ func standDownUnusedSelfCheck(opts option.Options) int {
|
||||
utOpts.SelfCheck = &off
|
||||
stood++
|
||||
}
|
||||
return stood
|
||||
return stood, used
|
||||
}
|
||||
|
||||
@@ -29,10 +29,17 @@ func TestStandDownUnusedSelfCheck(t *testing.T) {
|
||||
Route: fixRoute("auto"),
|
||||
}
|
||||
|
||||
stood := standDownUnusedSelfCheck(opts)
|
||||
stood, used := standDownUnusedSelfCheck(opts)
|
||||
if stood != 1 {
|
||||
t.Fatalf("stood down %d group(s), want exactly 1 (idle)", stood)
|
||||
}
|
||||
// The returned used-set is the SAME walk the flags were written from — the
|
||||
// engine publishes it for ProbeWhenIdle before the new box is built, so the
|
||||
// two statements about one group ("keeps its self-check", "keeps it warm")
|
||||
// can never be computed from two different reachability answers.
|
||||
if !used["auto"] || used["idle"] {
|
||||
t.Errorf("used-set = %v, want the reached group in and the unreached one out", used)
|
||||
}
|
||||
|
||||
find := func(tag string) option.Outbound {
|
||||
t.Helper()
|
||||
@@ -70,7 +77,7 @@ func TestStandDownUnusedSelfCheck(t *testing.T) {
|
||||
// still counted — the function reports plan membership, not novelty) and
|
||||
// changes nothing further. This is what keeps the apply hash stable across
|
||||
// the every-minute reconcile.
|
||||
if again := standDownUnusedSelfCheck(opts); again != 1 {
|
||||
if again, _ := standDownUnusedSelfCheck(opts); again != 1 {
|
||||
t.Fatalf("second pass stood down %d, want 1 (deterministic on identical opts)", again)
|
||||
}
|
||||
if idle.SelfCheck == nil || *idle.SelfCheck {
|
||||
@@ -144,10 +151,23 @@ func TestStandDownNeverTouchesChainHopWrappers(t *testing.T) {
|
||||
Route: fixRoute("", "chain-p-h4"),
|
||||
}
|
||||
|
||||
stood := standDownUnusedSelfCheck(opts)
|
||||
stood, used := standDownUnusedSelfCheck(opts)
|
||||
if stood != 3 {
|
||||
t.Fatalf("stood down %d group(s), want exactly the 3 base groups sub0/sub1/sub2", stood)
|
||||
}
|
||||
// The hop wrappers are also what ProbeWhenIdle must say yes to: the chain is
|
||||
// reached by a rule, so its hops keep measuring whether or not traffic is
|
||||
// crossing them. The base groups behind them are not, and must not.
|
||||
for _, tag := range []string{"chain-p-h2", "chain-p-h3", "chain-p-h4"} {
|
||||
if !used[tag] {
|
||||
t.Errorf("used-set is missing hop wrapper %q; it would idle out with nothing to replace it", tag)
|
||||
}
|
||||
}
|
||||
for _, tag := range []string{"sub0", "sub1", "sub2"} {
|
||||
if used[tag] {
|
||||
t.Errorf("used-set claims base group %q, which no rule reaches", tag)
|
||||
}
|
||||
}
|
||||
|
||||
utOptions := func(tag string) *option.URLTestOutboundOptions {
|
||||
t.Helper()
|
||||
|
||||
Reference in New Issue
Block a user