fix(wireguard): stop holding AmneziaWG down behind a WireGuard hop

The guard refused to start an AmneziaWG endpoint whose detour chain reached a
WireGuard one, and refused silently: not an error, just started=false, after
which every dial failed with "WireGuard is not ready yet". A selector hook went
further and suspended an already-working node the moment its group switched to a
WireGuard member.

It existed because AmneziaWG inside WireGuard hung the kernel on Android. We do
not ship Android, upstream dropped the guard once the cause was gone, and the
cure landed here yesterday — the ClientBind reserved-gate plus the submodule pin
that carries its twin. So the tree held both the cure and the prohibition on
using it, and the configuration simply did not come up while looking like a node
that "just does not work".

Also takes the two fixes that belong with it. ClientBind.conn was read on a
lock-free fast path and written under a mutex; upstream found that race with the
same end-to-end test we wrote yesterday, so we had taken one half of a pair
again. And the outer WireGuard UDP socket forced DF, unlike direct, hysteria and
tuic — with encapsulation the datagram regularly exceeds the path MTU and the
kernel drops it instead of fragmenting, a symptom indistinguishable from the bug
we spent yesterday on.

The race needed its own test: the existing e2e run did not flag it under -race
even at -count=15. Eight goroutines over both connect branches reproduce it
deterministically, naming the lock-free read and the guarded write.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-26 04:53:32 +03:00
co-authored by Claude Opus 5
parent 56a276bcc1
commit daaa0fda41
17 changed files with 513 additions and 408 deletions
@@ -3,7 +3,33 @@
| Поле | Значение |
|------|----------|
| Тип | B (bug) |
| Статус | C (complete) |
| Статус | C (complete) — guard **снят** (см. баннер ниже) |
> ## ⛔️ Guard снят (2026-07-26) — первопричина к shater не относится
> **Оба guard'а (Start-guard в `protocol/wireguard/endpoint.go` и
> selector-guard в `protocol/group/awg_selector_guard.go`) удалены**, вместе с
> их adapter-хуками (`OutboundManager.ConsumersOf`, `AmneziaWGSuspendable`).
> Апстрим снял их коммитом `5fa3a0a17`; сюда снятие приехало отдельно.
>
> **Почему.** Зависание было **Android-специфичным** (`Libbox.newService` не
> возвращал управление). Android для shater не платформа и ей не станет —
> мы собираем роутерный бинарь под OpenWrt/aarch64. При этом лекарство для
> самой AWG-за-detour связки у нас уже есть: reserved-clear gate в
> `ClientBind` (`d971eb85e` + пин сабмодуля `7d15f33`), без которого AWG не
> поднимался вообще ни за каким detour'ом. Мы носили и лекарство, и запрет
> на его применение.
>
> **Чем это было плохо на практике.** Guard отказывал **молча**: не ошибкой,
> а `started=false`, после чего каждый дозвон падал с «WireGuard is not ready
> yet». Конфигурация «AmneziaWG за WireGuard-хопом» выглядела не как
> отклонённая, а как «нода почему-то не работает».
>
> **Регрессия:** `protocol/wireguard/awg_over_wireguard_start_lx_test.go`
> (`with_gvisor && with_awg`) — AWG-эндпоинт с `detour` на outbound типа
> `wireguard` доходит до PostStart и поднимает `started`. До снятия guard'а
> тест краснел.
>
> **Осталось:** сквозной прогон на железе (AWG поверх реального WG-хопа).
Отклонять (по образцу ядрового запрета «empty direct detour») конфигурацию, где
AmneziaWG-endpoint (источник с AWG-полями) имеет `detour` на **любой
-22
View File
@@ -45,30 +45,8 @@ type OutboundManager interface {
Default() Outbound
Remove(tag string) error
Create(ctx context.Context, router Router, logger log.ContextLogger, tag string, outboundType string, options any) error
// lx:begin awg
// ConsumersOf returns the tags of outbounds that depend on (detour through)
// the given tag — the reverse of Dependencies(). Used by the selector guard to
// walk up to AmneziaWG consumers when a group switches to a WireGuard member.
ConsumersOf(tag string) []string
// lx:end awg
}
// lx:begin awg
// AmneziaWGSuspendable is implemented by an AmneziaWG endpoint so the selector
// guard can suspend it (bring its device down) when a group it detours through
// switches to a WireGuard member — AmneziaWG inside a WireGuard tunnel hangs the
// kernel on Android. The marker lives in adapter so protocol/group can act on it
// without importing protocol/wireguard.
type AmneziaWGSuspendable interface {
// IsAmneziaWG reports whether this endpoint runs AmneziaWG (has AWG params).
IsAmneziaWG() bool
// SuspendAmneziaWG brings the device down so no junk handshake is sent. It is
// idempotent and safe to call on a not-yet-started or already-suspended endpoint.
SuspendAmneziaWG()
}
// lx:end awg
// lx:begin idle-suspend
// IdleSuspendable is implemented by a WG/AWG endpoint so the router's idle tick
// (SPEC 020) can suspend it when it is idle and unreachable, without importing
-15
View File
@@ -208,21 +208,6 @@ func (m *Manager) Outbound(tag string) (adapter.Outbound, bool) {
return m.endpoint.Get(tag)
}
// lx:begin awg
// ConsumersOf returns a copy of the tags that detour through tag (reverse of
// Dependencies()), built from the dependByTag ledger populated at Create time.
func (m *Manager) ConsumersOf(tag string) []string {
m.access.RLock()
defer m.access.RUnlock()
consumers := m.dependByTag[tag]
if len(consumers) == 0 {
return nil
}
return append([]string(nil), consumers...)
}
// lx:end awg
func (m *Manager) Default() adapter.Outbound {
m.access.RLock()
defer m.access.RUnlock()
@@ -0,0 +1,35 @@
//go:build darwin
package dialer
import (
"syscall"
"testing"
"golang.org/x/sys/unix"
)
// udpSocketDFSet reports whether the socket has "don't fragment" forced on
// (control.DisableUDPFragment sets IP_DONTFRAG=1 on darwin).
func udpSocketDFSet(t *testing.T, sysConn syscall.Conn) bool {
t.Helper()
rawConn, err := sysConn.SyscallConn()
if err != nil {
t.Fatal(err)
}
var (
value int
sockErr error
ctrlErr error
)
ctrlErr = rawConn.Control(func(fd uintptr) {
value, sockErr = unix.GetsockoptInt(int(fd), unix.IPPROTO_IP, unix.IP_DONTFRAG)
})
if ctrlErr != nil {
t.Fatal(ctrlErr)
}
if sockErr != nil {
t.Fatal(sockErr)
}
return value != 0
}
@@ -0,0 +1,36 @@
//go:build linux
package dialer
import (
"syscall"
"testing"
"golang.org/x/sys/unix"
)
// udpSocketDFSet reports whether the socket has "don't fragment" forced on
// (control.DisableUDPFragment sets IP_MTU_DISCOVER=IP_PMTUDISC_DO on linux,
// the same flag the user-visible failure was traced to on android).
func udpSocketDFSet(t *testing.T, sysConn syscall.Conn) bool {
t.Helper()
rawConn, err := sysConn.SyscallConn()
if err != nil {
t.Fatal(err)
}
var (
value int
sockErr error
ctrlErr error
)
ctrlErr = rawConn.Control(func(fd uintptr) {
value, sockErr = unix.GetsockoptInt(int(fd), unix.IPPROTO_IP, unix.IP_MTU_DISCOVER)
})
if ctrlErr != nil {
t.Fatal(ctrlErr)
}
if sockErr != nil {
t.Fatal(sockErr)
}
return value == unix.IP_PMTUDISC_DO
}
@@ -0,0 +1,14 @@
//go:build !darwin && !linux && !windows
package dialer
import (
"syscall"
"testing"
)
func udpSocketDFSet(t *testing.T, _ syscall.Conn) bool {
t.Helper()
t.Skip("DF socket-flag introspection implemented for darwin, linux and windows only")
return false
}
@@ -0,0 +1,43 @@
//go:build windows
package dialer
import (
"syscall"
"testing"
"golang.org/x/sys/windows"
)
// IP_MTU_DISCOVER on windows (ws2ipdef.h); control.DisableUDPFragment sets it to
// IP_PMTUDISC_DO, the same "don't fragment" state the linux helper checks.
const (
windowsIPMTUDiscover = 71
windowsPMTUDiscDo = 1
)
// udpSocketDFSet reports whether the socket has "don't fragment" forced on.
// shater addition: upstream ships linux + darwin only, so the whole suite
// skipped on the dev host — where it is the one platform we can actually run it
// on before the router build.
func udpSocketDFSet(t *testing.T, sysConn syscall.Conn) bool {
t.Helper()
rawConn, err := sysConn.SyscallConn()
if err != nil {
t.Fatal(err)
}
var (
value int
sockErr error
)
ctrlErr := rawConn.Control(func(fd uintptr) {
value, sockErr = windows.GetsockoptInt(windows.Handle(fd), windows.IPPROTO_IP, windowsIPMTUDiscover)
})
if ctrlErr != nil {
t.Fatal(ctrlErr)
}
if sockErr != nil {
t.Skip("IP_MTU_DISCOVER is not readable on this host: ", sockErr)
}
return value == windowsPMTUDiscDo
}
+99
View File
@@ -0,0 +1,99 @@
// lx: regression tests for the udp_fragment / UDPFragmentDefault
// plumbing. The WireGuard endpoint (and MASQUE outbound) rely on
// UDPFragmentDefault=true reaching the real UDP socket as "DF clear": with DF
// set, an outer datagram larger than the path MTU is silently dropped instead
// of fragmented, which blackholes nested tunnels (AWG-over-AWG, MASQUE-over-AWG)
// and AWG s4 transport junk. These tests assert the socket flag itself, on both
// paths a WireGuard bind can take: the dialer (ClientBind, detour case) and the
// listener control (StdNetBind via WireGuardControl, no-detour case).
package dialer
import (
"context"
"net"
"syscall"
"testing"
"github.com/sagernet/sing-box/option"
M "github.com/sagernet/sing/common/metadata"
N "github.com/sagernet/sing/common/network"
)
func dialUDPForDF(t *testing.T, options option.DialerOptions) syscall.Conn {
t.Helper()
d, err := NewDefault(context.Background(), options)
if err != nil {
t.Fatal(err)
}
conn, err := d.DialContext(context.Background(), N.NetworkUDP, M.ParseSocksaddr("127.0.0.1:9"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = conn.Close() })
sysConn, isSysConn := conn.(syscall.Conn)
if !isSysConn {
t.Fatalf("dialed UDP conn %T does not expose SyscallConn", conn)
}
return sysConn
}
func listenUDPForDF(t *testing.T, options option.DialerOptions) syscall.Conn {
t.Helper()
d, err := NewDefault(context.Background(), options)
if err != nil {
t.Fatal(err)
}
// WireGuardControl() is the listener control conn.StdNetBind installs on the
// socket a no-detour WireGuard endpoint sends its outer datagrams from — the
// exact socket the DF default decides the fate of.
listenConfig := net.ListenConfig{Control: d.WireGuardControl()}
packetConn, err := listenConfig.ListenPacket(context.Background(), "udp4", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = packetConn.Close() })
sysConn, isSysConn := packetConn.(syscall.Conn)
if !isSysConn {
t.Fatalf("listened UDP conn %T does not expose SyscallConn", packetConn)
}
return sysConn
}
// Upstream default: no UDPFragmentDefault, no udp_fragment → DF is set on both
// the dial and listener paths. Pins the baseline the endpoint fix opts out of.
func TestUDPFragmentDFByDefault_LX(t *testing.T) {
if !udpSocketDFSet(t, dialUDPForDF(t, option.DialerOptions{})) {
t.Fatal("default dialer must set DF on dialed UDP sockets")
}
if !udpSocketDFSet(t, listenUDPForDF(t, option.DialerOptions{})) {
t.Fatal("default dialer must set DF on listener-control UDP sockets")
}
}
// UDPFragmentDefault=true (what the WireGuard endpoint and MASQUE outbound now
// set) → DF clear on both paths, so oversize outer datagrams fragment instead
// of vanishing.
func TestUDPFragmentDefaultClearsDF_LX(t *testing.T) {
options := option.DialerOptions{UDPFragmentDefault: true}
if udpSocketDFSet(t, dialUDPForDF(t, options)) {
t.Fatal("UDPFragmentDefault=true must leave DF clear on dialed UDP sockets")
}
if udpSocketDFSet(t, listenUDPForDF(t, options)) {
t.Fatal("UDPFragmentDefault=true must leave DF clear on listener-control UDP sockets")
}
}
// Explicit user config always wins over the protocol default, in both
// directions.
func TestUDPFragmentExplicitOverride_LX(t *testing.T) {
fragmentOff := false
options := option.DialerOptions{UDPFragment: &fragmentOff, UDPFragmentDefault: true}
if !udpSocketDFSet(t, dialUDPForDF(t, options)) {
t.Fatal("udp_fragment=false must set DF even when the protocol default allows fragmentation")
}
fragmentOn := true
options = option.DialerOptions{UDPFragment: &fragmentOn}
if udpSocketDFSet(t, dialUDPForDF(t, options)) {
t.Fatal("udp_fragment=true must leave DF clear even without a protocol default")
}
}
-93
View File
@@ -1,93 +0,0 @@
// lx:begin awg
package group
import (
"github.com/sagernet/sing-box/adapter"
C "github.com/sagernet/sing-box/constant"
)
// suspendAmneziaWGConsumersOnWireGuardSwitch is called from Selector.SelectOutbound
// BEFORE the switch is committed. If the member about to be selected is — or chains
// down via detour to — a WireGuard-based endpoint (type "wireguard", covering plain
// WG and AmneziaWG), it walks UP from this group to every AmneziaWG endpoint that
// detours through it and suspends each one (brings its device down). Rationale:
// AmneziaWG traffic encapsulated inside a WireGuard tunnel hangs the kernel on
// Android; the static Start-guard cannot cover this because a selector's chosen
// member is only known at runtime.
//
// Called before s.selected is updated, so the race is closed: once the group
// points at the WireGuard member, the AmneziaWG consumers are already suspended
// (started=false) and a concurrent reconnect fails with "not ready" instead of
// sending a junk handshake into WireGuard.
func suspendAmneziaWGConsumersOnWireGuardSwitch(outboundManager adapter.OutboundManager, groupTag string, selected adapter.Outbound) {
if outboundManager == nil || groupTag == "" {
return
}
if !chainReachesWireGuard(outboundManager, selected, make(map[string]bool)) {
return
}
suspendAmneziaWGConsumers(outboundManager, groupTag, make(map[string]bool))
}
// chainReachesWireGuard reports whether outbound is — or transitively detours
// down to, or (being a group) contains a member that is — a WireGuard-based
// endpoint. visited guards against cycles.
func chainReachesWireGuard(outboundManager adapter.OutboundManager, outbound adapter.Outbound, visited map[string]bool) bool {
if outbound == nil {
return false
}
tag := outbound.Tag()
if tag != "" {
if visited[tag] {
return false
}
visited[tag] = true
}
if outbound.Type() == C.TypeWireGuard {
return true
}
// Down the detour chain (vless -> ... -> wireguard).
for _, dependency := range outbound.Dependencies() {
if member, loaded := outboundManager.Outbound(dependency); loaded {
if chainReachesWireGuard(outboundManager, member, visited) {
return true
}
}
}
// A nested group: any member reaching WireGuard counts.
if group, isGroup := outbound.(adapter.OutboundGroup); isGroup {
for _, memberTag := range group.All() {
if member, loaded := outboundManager.Outbound(memberTag); loaded {
if chainReachesWireGuard(outboundManager, member, visited) {
return true
}
}
}
}
return false
}
// suspendAmneziaWGConsumers walks UP from tag via the reverse-dependency ledger
// (ConsumersOf) and suspends every AmneziaWG endpoint that detours through it,
// directly or transitively (e.g. AWG -> vless -> group). visited guards cycles.
func suspendAmneziaWGConsumers(outboundManager adapter.OutboundManager, tag string, visited map[string]bool) {
for _, consumerTag := range outboundManager.ConsumersOf(tag) {
if visited[consumerTag] {
continue
}
visited[consumerTag] = true
consumer, loaded := outboundManager.Outbound(consumerTag)
if !loaded {
continue
}
if awg, isAWG := consumer.(adapter.AmneziaWGSuspendable); isAWG && awg.IsAmneziaWG() {
awg.SuspendAmneziaWG()
}
// Keep walking up: a non-AWG hop (vless) or a parent group may itself have
// an AmneziaWG consumer above it.
suspendAmneziaWGConsumers(outboundManager, consumerTag, visited)
}
}
// lx:end awg
-120
View File
@@ -1,120 +0,0 @@
// lx:begin awg
package group
import (
"testing"
"github.com/sagernet/sing-box/adapter"
C "github.com/sagernet/sing-box/constant"
)
// fakeOutbound is a minimal adapter.Outbound; only Type/Tag/Dependencies are read.
type fakeOutbound struct {
adapter.Outbound
tag string
outboundTyp string
detour string
}
func (o *fakeOutbound) Type() string { return o.outboundTyp }
func (o *fakeOutbound) Tag() string { return o.tag }
func (o *fakeOutbound) Dependencies() []string {
if o.detour == "" {
return nil
}
return []string{o.detour}
}
// fakeAWG implements adapter.AmneziaWGSuspendable and records suspension.
type fakeAWG struct {
fakeOutbound
awg bool
suspended bool
}
func (a *fakeAWG) IsAmneziaWG() bool { return a.awg }
func (a *fakeAWG) SuspendAmneziaWG() { a.suspended = true }
// fakeManager resolves tags and reverse-deps (ConsumersOf) from fixed maps.
type fakeManager struct {
adapter.OutboundManager
byTag map[string]adapter.Outbound
consumers map[string][]string
}
func (m *fakeManager) Outbound(tag string) (adapter.Outbound, bool) {
ob, ok := m.byTag[tag]
return ob, ok
}
func (m *fakeManager) ConsumersOf(tag string) []string { return m.consumers[tag] }
func TestChainReachesWireGuard(t *testing.T) {
wg := &fakeOutbound{tag: "wg", outboundTyp: C.TypeWireGuard}
vlessToWG := &fakeOutbound{tag: "v2wg", outboundTyp: C.TypeVLESS, detour: "wg"}
vlessLeaf := &fakeOutbound{tag: "vleaf", outboundTyp: C.TypeVLESS}
mgr := &fakeManager{byTag: map[string]adapter.Outbound{
"wg": wg, "v2wg": vlessToWG, "vleaf": vlessLeaf,
}}
if !chainReachesWireGuard(mgr, wg, map[string]bool{}) {
t.Fatal("direct wireguard member must reach wireguard")
}
if !chainReachesWireGuard(mgr, vlessToWG, map[string]bool{}) {
t.Fatal("vless detouring to wireguard must reach wireguard")
}
if chainReachesWireGuard(mgr, vlessLeaf, map[string]bool{}) {
t.Fatal("plain vless must not reach wireguard")
}
}
func TestSuspendAmneziaWGConsumers(t *testing.T) {
// awg-direct detours through the group "sel"
awgDirect := &fakeAWG{fakeOutbound: fakeOutbound{tag: "awg-direct", outboundTyp: C.TypeWireGuard, detour: "sel"}, awg: true}
// awg-via-hop -> vless-hop -> sel
awgViaHop := &fakeAWG{fakeOutbound: fakeOutbound{tag: "awg-hop", outboundTyp: C.TypeWireGuard, detour: "vless-hop"}, awg: true}
vlessHop := &fakeOutbound{tag: "vless-hop", outboundTyp: C.TypeVLESS, detour: "sel"}
// plain-wg detours through sel but is NOT amneziawg — must stay untouched
plainWG := &fakeAWG{fakeOutbound: fakeOutbound{tag: "plain-wg", outboundTyp: C.TypeWireGuard, detour: "sel"}, awg: false}
mgr := &fakeManager{
byTag: map[string]adapter.Outbound{
"awg-direct": awgDirect, "awg-hop": awgViaHop,
"vless-hop": vlessHop, "plain-wg": plainWG,
},
consumers: map[string][]string{
"sel": {"awg-direct", "vless-hop", "plain-wg"},
"vless-hop": {"awg-hop"},
},
}
suspendAmneziaWGConsumers(mgr, "sel", map[string]bool{})
if !awgDirect.suspended {
t.Error("direct AmneziaWG consumer must be suspended")
}
if !awgViaHop.suspended {
t.Error("transitive AmneziaWG consumer (via vless hop) must be suspended")
}
if plainWG.suspended {
t.Error("plain (non-AmneziaWG) wireguard consumer must NOT be suspended")
}
}
// A switch to a non-wireguard member must suspend nothing.
func TestSuspendSkippedForNonWireGuardSwitch(t *testing.T) {
awg := &fakeAWG{fakeOutbound: fakeOutbound{tag: "awg", outboundTyp: C.TypeWireGuard, detour: "sel"}, awg: true}
vlessLeaf := &fakeOutbound{tag: "vleaf", outboundTyp: C.TypeVLESS}
mgr := &fakeManager{
byTag: map[string]adapter.Outbound{"awg": awg, "vleaf": vlessLeaf},
consumers: map[string][]string{"sel": {"awg"}},
}
// selected member is plain vless (does not reach wireguard) → no suspension
suspendAmneziaWGConsumersOnWireGuardSwitch(mgr, "sel", vlessLeaf)
if awg.suspended {
t.Error("must not suspend when the selected member does not reach wireguard")
}
}
// lx:end awg
-10
View File
@@ -128,16 +128,6 @@ func (s *Selector) SelectOutbound(tag string) bool {
if s.selected.Load() == detour {
return true
}
// lx:begin awg
// Suspend AmneziaWG consumers BEFORE switching: if the new member is (or chains
// to) a WireGuard endpoint, any AmneziaWG endpoint that detours through this
// group would tunnel AWG inside WireGuard and hang the kernel on Android. Doing
// this before s.selected.Swap closes the race — by the time the group points at
// the WireGuard member, those consumers are already down (started=false), so a
// concurrent reconnect fails with "not ready" instead of sending a junk
// handshake into WireGuard.
suspendAmneziaWGConsumersOnWireGuardSwitch(s.outbound, s.Tag(), detour)
// lx:end awg
s.selected.Store(detour)
invalidateReachability(s.ctx) // lx: SPEC 020 — active selection changed
if s.Tag() != "" {
@@ -0,0 +1,129 @@
//go:build with_gvisor && with_awg
// lx: regression for the removal of the AmneziaWG-over-WireGuard start guard.
//
// The guard refused to bring up an AmneziaWG endpoint whose detour chain reached
// a WireGuard-based endpoint — and refused *silently*: Start returned nil with
// started=false, so the endpoint looked configured but every dial through it
// failed with "WireGuard is not ready yet". The root cause it protected against
// (a kernel hang on Android) is gone on this graft (ClientBind reserved-gate),
// and Android is not a supported platform here at all.
//
// This test builds a real AmneziaWG endpoint (junk + ranged magic headers) whose
// detour points at an outbound of type "wireguard", drives both start stages,
// and asserts the endpoint reports itself started. With the guard in place the
// first stage short-circuits and started stays false — this test fails.
package wireguard
import (
"context"
"crypto/rand"
"encoding/base64"
"net"
"net/netip"
"os"
"testing"
"github.com/sagernet/sing-box/adapter"
C "github.com/sagernet/sing-box/constant"
"github.com/sagernet/sing-box/log"
"github.com/sagernet/sing-box/option"
"github.com/sagernet/sing/common/json/badoption"
M "github.com/sagernet/sing/common/metadata"
"github.com/sagernet/sing/service"
"github.com/sagernet/sing/service/pause"
)
// wgTypedOutbound is an adapter.Outbound that reports type "wireguard" — the hop
// the guard used to refuse to start behind. Dialling through it always fails:
// the point of the test is that the upper endpoint comes UP, not that it carries
// traffic (that is the job of the transport-level e2e stand).
type wgTypedOutbound struct {
adapter.Outbound
tag string
}
func (o *wgTypedOutbound) Type() string { return C.TypeWireGuard }
func (o *wgTypedOutbound) Tag() string { return o.tag }
func (o *wgTypedOutbound) Dependencies() []string { return nil }
func (o *wgTypedOutbound) DialContext(ctx context.Context, network string, destination M.Socksaddr) (net.Conn, error) {
return nil, os.ErrClosed
}
func (o *wgTypedOutbound) ListenPacket(ctx context.Context, destination M.Socksaddr) (net.PacketConn, error) {
return nil, os.ErrClosed
}
// startChainManager resolves tags from a fixed map. adapter.OutboundManager is
// embedded so this compiles against either shape of the interface.
type startChainManager struct {
adapter.OutboundManager
byTag map[string]adapter.Outbound
}
func (m *startChainManager) Outbound(tag string) (adapter.Outbound, bool) {
ob, loaded := m.byTag[tag]
return ob, loaded
}
func randomKey(t *testing.T) string {
t.Helper()
var key [32]byte
if _, err := rand.Read(key[:]); err != nil {
t.Fatal(err)
}
// Clamp so wireguard-go accepts it as a curve25519 private key.
key[0] &= 248
key[31] = (key[31] & 127) | 64
return base64.StdEncoding.EncodeToString(key[:])
}
// TestAmneziaWGOverWireGuardDetourStarts pins the invariant: an AmneziaWG
// endpoint detouring through a WireGuard hop must come up like any other.
func TestAmneziaWGOverWireGuardDetourStarts(t *testing.T) {
ctx := pause.WithDefaultManager(context.Background())
ctx = service.ContextWith[adapter.OutboundManager](ctx, &startChainManager{
byTag: map[string]adapter.Outbound{
"wg-hop": &wgTypedOutbound{tag: "wg-hop"},
},
})
options := option.WireGuardEndpointOptions{
MTU: 1280,
Address: badoption.Listable[netip.Prefix]{netip.MustParsePrefix("10.7.0.2/32")},
PrivateKey: randomKey(t),
Peers: []option.WireGuardPeer{{
Address: "10.9.9.9",
Port: 51820,
PublicKey: randomKey(t),
AllowedIPs: badoption.Listable[netip.Prefix]{netip.MustParsePrefix("0.0.0.0/0")},
}},
AmneziaWGOptions: option.AmneziaWGOptions{
Jc: 3,
Jmin: 8,
Jmax: 80,
S4: 16,
H1: "10-20",
H2: "30-40",
H3: "50-60",
H4: "70-80",
},
}
options.Detour = "wg-hop"
ep, err := NewEndpoint(ctx, nil, log.NewNOPFactory().NewLogger("wg-awg"), "wg-awg", options)
if err != nil {
t.Fatal("create amneziawg endpoint over a wireguard detour: ", err)
}
defer ep.Close()
if err = ep.Start(adapter.StartStateStart); err != nil {
t.Fatal("start stage: ", err)
}
if err = ep.Start(adapter.StartStatePostStart); err != nil {
t.Fatal("post-start stage: ", err)
}
if !ep.(*Endpoint).started.Load() {
t.Fatal("an amneziawg endpoint behind a wireguard hop must start; it is silently held down")
}
}
+20 -118
View File
@@ -4,7 +4,6 @@ import (
"context"
"net"
"net/netip"
"strconv"
"sync"
"sync/atomic"
"time"
@@ -45,28 +44,14 @@ type Endpoint struct {
localAddresses []netip.Prefix
endpoint *wireguard.Endpoint
started atomic.Bool
// lx:begin awg
// awgActive marks this endpoint as running AmneziaWG (AmneziaWGOptions.IsSet());
// detour is its configured upstream tag. Start uses them to refuse to bring up
// an AmneziaWG-over-WireGuard chain, which hangs the kernel on Android — see
// awgDetourChainReachesWireGuard. The ledger lives here (not just in the dialer
// guard) because the hang happens synchronously in Start, before any dial.
awgActive bool
detour string
// awgChainBlocked is set by Start when the AmneziaWG-over-WireGuard guard
// fires: the device is left unstarted (started stays false) so no junk
// handshake runs and the kernel cannot hang, while the rest of the instance
// comes up. PostStart then skips this endpoint too.
awgChainBlocked bool
// lx:end awg
// lx:begin idle-suspend
// SPEC 020 idle-suspend state. lastActivity is the unix-nano timestamp of the
// last dial through this endpoint, stamped at PostStart and on every dial entry.
// idleAsleep is true while the endpoint is Down due to idle-suspend (distinct
// from a guard-suspend, which sets started=false and clears idleAsleep, so a
// guard-suspended endpoint fast-paths out of resumeOnDial and is never
// idle-woken). resumeMu serialises the idle tick's suspend decision, a dial's
// wake, and the AmneziaWG guard-suspend against one another.
// from a deliberately-stopped endpoint, which has started=false and
// idleAsleep=false, so it fast-paths out of resumeOnDial and is never
// idle-woken). resumeMu serialises the idle tick's suspend decision against a
// dial's wake.
lastActivity atomic.Int64
idleAsleep atomic.Bool
resumeMu sync.Mutex
@@ -74,6 +59,16 @@ type Endpoint struct {
}
func NewEndpoint(ctx context.Context, router adapter.Router, logger log.ContextLogger, tag string, options option.WireGuardEndpointOptions) (adapter.Endpoint, error) {
// lx: allow OS-level fragmentation of the OUTER UDP socket by default, the
// same opt-out direct/hysteria/hysteria2/tuic already take. Without it the
// dialer sets DF (IP_MTU_DISCOVER=IP_PMTUDISC_DO on linux), and an outer
// datagram over the path MTU — routine once anything is encapsulated: WG's
// own ~32 B header, AmneziaWG s4 transport junk, or this endpoint carrying a
// nested tunnel — is dropped by the kernel ("message too long") instead of
// fragmented, so the tunnel comes up and then carries nothing. An explicit
// `udp_fragment: false` on the node still restores DF (UDPFragment wins over
// UDPFragmentDefault in common/dialer).
options.UDPFragmentDefault = true
ep := &Endpoint{
Adapter: endpoint.NewAdapterWithDialerOptions(C.TypeWireGuard, tag, []string{N.NetworkTCP, N.NetworkUDP, N.NetworkICMP}, options.DialerOptions),
ctx: ctx,
@@ -81,10 +76,6 @@ func NewEndpoint(ctx context.Context, router adapter.Router, logger log.ContextL
dnsRouter: service.FromContext[adapter.DNSRouter](ctx),
logger: logger,
localAddresses: options.Address,
// lx:begin awg
awgActive: options.AmneziaWGOptions.IsSet(),
detour: options.Detour,
// lx:end awg
}
if options.Detour != "" && options.ListenPort != 0 {
return nil, E.New("`listen_port` is conflict with `detour`")
@@ -116,7 +107,8 @@ func NewEndpoint(ctx context.Context, router adapter.Router, logger log.ContextL
Dialer: outboundDialer,
CreateDialer: func(interfaceName string) N.Dialer {
return common.Must1(dialer.NewDefault(ctx, option.DialerOptions{
BindInterface: interfaceName,
BindInterface: interfaceName,
UDPFragmentDefault: true, // lx: same reason as above — this is the bind-to-interface twin of the outer socket
}))
},
Name: options.Name,
@@ -157,33 +149,6 @@ func NewEndpoint(ctx context.Context, router adapter.Router, logger log.ContextL
}
func (w *Endpoint) Start(stage adapter.StartStage) error {
// lx:begin awg
// Refuse to bring up an AmneziaWG endpoint whose detour chain reaches a
// WireGuard-based endpoint: encapsulating AWG (junk handshake) inside a
// WireGuard tunnel hangs the kernel on Android. The hang happens here, in the
// synchronous Start path (peer-domain resolution over the detour, then the
// device's junk handshake) — before any dial — so the lazy DetourDialer guard
// never gets a chance to fire. We must catch it at Start instead.
//
// Behaviour is "variant B": do NOT return an error (that would abort the whole
// instance start). Instead log, skip device startup, and leave started=false
// so the rest of the config comes up and every dial through this endpoint
// fails cleanly with "WireGuard is not ready yet". A selector/urltest in the
// middle hides the real target at start time, so the chain walk stops at a
// group and that case is left to the lazy DetourDialer guard at dial time.
if stage == adapter.StartStateStart && w.awgActive && w.detour != "" {
if outboundManager := service.FromContext[adapter.OutboundManager](w.ctx); outboundManager != nil {
if blockedBy := awgDetourChainReachesWireGuard(outboundManager, w.detour, make(map[string]bool)); blockedBy != "" {
w.awgChainBlocked = true
w.logger.Error("amneziawg endpoint will not start: its detour chain reaches wireguard-based endpoint ", strconv.Quote(blockedBy), " — amneziawg over wireguard is not supported. Use a non-wireguard detour (e.g. vless).")
return nil
}
}
}
if w.awgChainBlocked {
return nil
}
// lx:end awg
switch stage {
case adapter.StartStateStart:
return w.endpoint.Start(false)
@@ -200,69 +165,6 @@ func (w *Endpoint) Start(stage adapter.StartStage) error {
return nil
}
// lx:begin awg
// awgDetourChainReachesWireGuard walks the transitive detour chain starting at
// tag and returns the tag of the first WireGuard-based outbound it reaches
// (type "wireguard", covering plain WireGuard and AmneziaWG), or "" if none. It
// follows each outbound's detour dependency; it deliberately does NOT expand
// selector/urltest groups, whose chosen member is only known at runtime — that
// case is handled lazily by the DetourDialer guard. visited guards against cyclic
// detour configs. All outbounds are registered before any Start, so every tag in
// the chain is resolvable here even though some may not have started yet.
func awgDetourChainReachesWireGuard(outboundManager adapter.OutboundManager, tag string, visited map[string]bool) string {
if tag == "" || visited[tag] {
return ""
}
visited[tag] = true
outbound, loaded := outboundManager.Outbound(tag)
if !loaded {
return ""
}
if outbound.Type() == C.TypeWireGuard {
return tag
}
if _, isGroup := outbound.(adapter.OutboundGroup); isGroup {
// Runtime-resolved target — leave it to the lazy DetourDialer guard.
return ""
}
for _, dependency := range outbound.Dependencies() {
if blockedBy := awgDetourChainReachesWireGuard(outboundManager, dependency, visited); blockedBy != "" {
return blockedBy
}
}
return ""
}
// IsAmneziaWG reports whether this endpoint runs AmneziaWG. Implements
// adapter.AmneziaWGSuspendable.
func (w *Endpoint) IsAmneziaWG() bool {
return w.awgActive
}
// SuspendAmneziaWG brings the device down and marks the endpoint not-ready, so a
// junk handshake is never sent and every dial fails with "WireGuard is not ready
// yet". Called by the selector guard when a group this endpoint detours through
// switches to a WireGuard member (AmneziaWG over WireGuard hangs the kernel on
// Android). Idempotent. Implements adapter.AmneziaWGSuspendable.
func (w *Endpoint) SuspendAmneziaWG() {
// Take resumeMu so this is ordered against resumeOnDial/SuspendIfIdle: without
// it, a dial that already passed resumeOnDial's idleAsleep checks could wake
// the endpoint back up right after we clear the flag, defeating the guard.
w.resumeMu.Lock()
defer w.resumeMu.Unlock()
if w.started.CompareAndSwap(true, false) {
w.logger.Error("amneziawg endpoint suspended: a selector in its detour chain switched to a wireguard-based member — amneziawg over wireguard is not supported")
}
// Clear any idle-suspend state so resumeOnDial does not resurrect a
// guard-suspended endpoint: if it was idle-asleep first, idleAsleep would still
// be true and the next dial would wake it (SPEC 022 #2). With idleAsleep=false
// resumeOnDial's fast path returns started (now false) and the endpoint stays down.
w.idleAsleep.Store(false)
w.endpoint.Suspend()
}
// lx:end awg
// lx:begin idle-suspend
// stampActivity records the current time as the last dial through this endpoint.
@@ -286,8 +188,8 @@ func (w *Endpoint) IdleSince() time.Duration {
// holder — when it is unreachable from the active routing tree AND has been idle
// past the threshold. Silent on every non-transition (edge-triggered logging).
//
// It never touches a guard-suspended endpoint: that one already has
// started==false but idleAsleep==false, and the `!started` guard below short-
// It never touches a deliberately-stopped endpoint: that one already has
// started==false but idleAsleep==false, and the `!started` check below short-
// circuits before the CAS. resumeMu mutually excludes this against resumeOnDial.
func (w *Endpoint) SuspendIfIdle(reachable bool, threshold time.Duration) {
w.resumeMu.Lock()
@@ -296,7 +198,7 @@ func (w *Endpoint) SuspendIfIdle(reachable bool, threshold time.Duration) {
return
}
if !w.started.Load() {
// Already down some other way (guard-suspend, awg-chain-blocked, closed).
// Already down some other way (deliberately stopped, closed).
return
}
if w.idleAsleep.CompareAndSwap(false, true) {
@@ -313,7 +215,7 @@ func (w *Endpoint) SuspendIfIdle(reachable bool, threshold time.Duration) {
// session); that cost is on the first packet, as for any cold WG dial.
//
// Returns true if the endpoint is dialable (awake), false if it must stay down
// (guard-suspend / chain-blocked — not an idle-suspend, so we do not resurrect it).
// (deliberately stopped / closed — not an idle-suspend, so we do not resurrect it).
func (w *Endpoint) resumeOnDial() bool {
w.stampActivity()
if !w.idleAsleep.Load() {
+15 -15
View File
@@ -103,21 +103,21 @@ func TestSuspendIfIdle_idempotentCAS(t *testing.T) {
}
}
// TestSuspendIfIdle_guardSuspendedNotTouched is the §8 invariant verified live on
// an AWG-over-WG endpoint (wg-3 in the prod run): a guard-suspended endpoint has
// started=false WITHOUT idleAsleep. The idle tick must early-return on !started and
// NOT flip idleAsleep — otherwise a later resumeOnDial would idle-wake it and
// re-trigger the AWG-over-WG kernel hang the guard exists to prevent.
func TestSuspendIfIdle_guardSuspendedNotTouched(t *testing.T) {
// TestSuspendIfIdle_stoppedNotTouched is the §8 invariant: a deliberately-stopped
// endpoint (Close, or a start that never completed) has started=false WITHOUT
// idleAsleep. The idle tick must early-return on !started and NOT flip idleAsleep
// — otherwise a later resumeOnDial would idle-wake a device that was
// intentionally down.
func TestSuspendIfIdle_stoppedNotTouched(t *testing.T) {
w := newIdleTestEndpoint()
w.started.Store(false) // guard-suspend (device.Down at Start), idleAsleep stays false
w.started.Store(false) // stopped, idleAsleep stays false
w.lastActivity.Store(time.Now().Add(-time.Hour).UnixNano())
w.SuspendIfIdle(false, 30*time.Second)
if w.idleAsleep.Load() {
t.Fatal("a guard-suspended endpoint must NOT be flagged idleAsleep by the tick")
t.Fatal("a stopped endpoint must NOT be flagged idleAsleep by the tick")
}
if w.started.Load() {
t.Fatal("the tick must not change started for a guard-suspended endpoint")
t.Fatal("the tick must not change started for a stopped endpoint")
}
}
@@ -152,17 +152,17 @@ func TestResumeOnDial_dialBeforeTickRace(t *testing.T) {
}
}
func TestResumeOnDial_guardSuspendedNotWoken(t *testing.T) {
// A guard-suspended endpoint has started=false but idleAsleep=false.
// resumeOnDial must NOT wake it (returns started, i.e. false).
func TestResumeOnDial_stoppedNotWoken(t *testing.T) {
// A deliberately-stopped endpoint (Close / failed start) has started=false but
// idleAsleep=false. resumeOnDial must NOT wake it (returns started, i.e. false).
w := newIdleTestEndpoint()
w.started.Store(false) // simulate guard/awg-chain suspend (not idle)
w.started.Store(false) // stopped, not idle-suspended
ok := w.resumeOnDial()
if ok {
t.Fatal("resumeOnDial must not resurrect a guard-suspended (non-idle) endpoint")
t.Fatal("resumeOnDial must not resurrect a stopped (non-idle) endpoint")
}
if w.idleAsleep.Load() {
t.Fatal("guard-suspended endpoint must not be flagged idleAsleep")
t.Fatal("stopped endpoint must not be flagged idleAsleep")
}
}
+17 -11
View File
@@ -5,6 +5,7 @@ import (
"net"
"net/netip"
"sync"
"sync/atomic"
"time"
"github.com/sagernet/sing/common"
@@ -29,11 +30,15 @@ type ClientBind struct {
dialer N.Dialer
reservedForEndpoint map[netip.AddrPort][3]uint8
connAccess sync.Mutex
conn *wireConn
done chan struct{}
isConnect bool
connectAddr netip.AddrPort
reserved [3]uint8
// lx: atomic — connect()'s fast-path read is lock-free while the store below
// happens under connAccess, so a plain field is a data race between the send
// and receive goroutines at tunnel start (upstream sing-box has the same
// shape). No logic change: the double-check and the locking are unchanged.
conn atomic.Pointer[wireConn]
done chan struct{}
isConnect bool
connectAddr netip.AddrPort
reserved [3]uint8
}
func NewClientBind(ctx context.Context, logger logger.Logger, dialer N.Dialer, isConnect bool, connectAddr netip.AddrPort, reserved [3]uint8) *ClientBind {
@@ -98,7 +103,7 @@ func (c *ClientBind) hasReserved() bool {
}
func (c *ClientBind) connect() (*wireConn, error) {
serverConn := c.conn
serverConn := c.conn.Load()
if serverConn != nil {
select {
case <-serverConn.done:
@@ -114,7 +119,7 @@ func (c *ClientBind) connect() (*wireConn, error) {
return nil, net.ErrClosed
default:
}
serverConn = c.conn
serverConn = c.conn.Load()
if serverConn != nil {
select {
case <-serverConn.done:
@@ -128,7 +133,7 @@ func (c *ClientBind) connect() (*wireConn, error) {
if err != nil {
return nil, err
}
c.conn = &wireConn{
serverConn = &wireConn{
PacketConn: bufio.NewUnbindPacketConn(udpConn),
done: make(chan struct{}),
}
@@ -137,12 +142,13 @@ func (c *ClientBind) connect() (*wireConn, error) {
if err != nil {
return nil, err
}
c.conn = &wireConn{
serverConn = &wireConn{
PacketConn: bufio.NewPacketConn(udpConn),
done: make(chan struct{}),
}
}
return c.conn, nil
c.conn.Store(serverConn)
return serverConn, nil
}
func (c *ClientBind) Open(port uint16) (fns []conn.ReceiveFunc, actualPort uint16, err error) {
@@ -206,7 +212,7 @@ func (c *ClientBind) Close() error {
}
c.connAccess.Lock()
defer c.connAccess.Unlock()
common.Close(common.PtrOrNil(c.conn))
common.Close(common.PtrOrNil(c.conn.Load()))
return nil
}
@@ -0,0 +1,76 @@
// lx: -race regression for ClientBind.conn.
//
// connect() reads the field on a lock-free fast path while the double-checked
// init writes it under connAccess. The send and receive goroutines both call
// connect() as the tunnel comes up, so the plain field was a data race (upstream
// sing-box has the same shape; fixed there by making it an atomic.Pointer).
// The e2e AWG-over-detour stand hit it only occasionally — this test drives the
// same two accesses deterministically, so `go test -race` sees them every run.
package wireguard
import (
"context"
"net"
"sync"
"testing"
"github.com/sagernet/sing/common/logger"
M "github.com/sagernet/sing/common/metadata"
N "github.com/sagernet/sing/common/network"
)
// rawUDPDialer hands out real (unconnected) UDP sockets on loopback, so
// connect() takes its normal path without any network peer.
type rawUDPDialer struct{}
func (rawUDPDialer) DialContext(ctx context.Context, network string, destination M.Socksaddr) (net.Conn, error) {
return net.Dial(N.NetworkUDP, destination.String())
}
func (rawUDPDialer) ListenPacket(ctx context.Context, destination M.Socksaddr) (net.PacketConn, error) {
return net.ListenPacket(N.NetworkUDP, "127.0.0.1:0")
}
func TestClientBindConnectConcurrent(t *testing.T) {
// A real listener so the "connect" dial has somewhere to point.
peer, err := net.ListenPacket(N.NetworkUDP, "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer peer.Close()
for _, isConnect := range []bool{true, false} {
bind := NewClientBind(
context.Background(),
logger.NOP(),
rawUDPDialer{},
isConnect,
M.SocksaddrFromNet(peer.LocalAddr()).Unwrap().AddrPort(),
[3]uint8{},
)
if _, _, err = bind.Open(0); err != nil {
t.Fatal(err)
}
// Eight goroutines racing the fast-path read against the locked store —
// exactly the send/receive pair at tunnel start, only wider.
var wg sync.WaitGroup
for range 8 {
wg.Add(1)
go func() {
defer wg.Done()
for range 32 {
if _, connErr := bind.connect(); connErr != nil {
return
}
}
}()
}
wg.Wait()
if bind.conn.Load() == nil {
t.Fatal("connect() must have installed a wireConn")
}
if err = bind.Close(); err != nil {
t.Fatal(err)
}
}
}
+2 -3
View File
@@ -278,9 +278,8 @@ func (e *Endpoint) Close() error {
// lx:begin awg
// Suspend brings the WireGuard device down without closing it, stopping the
// (junk) handshake machinery. Used by the selector guard to neutralise an
// AmneziaWG endpoint when a group it detours through switches to a WireGuard
// member. Idempotent — a nil device (never started / already closed) is a no-op.
// (junk) handshake machinery. Idempotent — a nil device (never started /
// already closed) is a no-op.
//
// SPEC 020 reuses Suspend for idle-suspend: device.Down() closes the UDP socket,
// which makes RoutineReceiveIncoming exit and release its bufsArrs (the dominant