feat(vp8channel): auto-discover wire rate via delay-based pacer

the hardcoded 400kb/s was a stability compromise and the previous knob
just moved the guesswork to operators: they had to hand-measure each
sfu's policer knee by ramping until stalls. replace both with a
delay-based aimd controller that finds the knee at runtime.

the sfu policer queues before it drops, so kcp's smoothed rtt inflates
ahead of the throughput collapse from #95/#107. the pacer folds one
srtt sample per writer tick: probe the rate up additively while delay
sits near the path baseline, back off multiplicatively once it
inflates. it settles just under the per-sfu knee on its own, both on
the client->server writerLoop and each server->client peer pump.

vp8.max_bytes_per_sec is now just the probe ceiling (default 1mb/s);
a fresh session still starts from the old 400kb/s operating point so a
healthy path only ramps up. no per-service hand-tuning left.

refs #107
This commit is contained in:
neuronori
2026-06-29 02:25:28 +00:00
parent 443f97edb0
commit 2dffb5acbe
10 changed files with 327 additions and 51 deletions
@@ -33,10 +33,10 @@ socks:
vp8:
fps: 30
batch_size: 64
# Потолок скорости на проводе (байт/сек). Дефолт 400000 (~400 КБ/с) -
# консервативно под Telemost. Подними под свой SFU, измерив стабильный
# максимум реальными запусками. 0 = дефолт.
# max_bytes_per_sec: 400000
# Потолок, до которого адаптивный пейсер может разгоняться (байт/сек).
# Скорость подбирается автоматически по задержке пути - под колено SFU.
# Дефолт 1000000 (~1 МБ/с). Опусти, чтобы жёстко лимитировать. 0 = дефолт.
# max_bytes_per_sec: 1000000
data: data
debug: false
@@ -34,10 +34,10 @@ socks:
vp8:
fps: 30
batch_size: 64
# Потолок скорости на проводе (байт/сек). Дефолт 400000 (~400 КБ/с) -
# консервативно под Telemost. Подними под свой SFU, измерив стабильный
# максимум реальными запусками. 0 = дефолт.
# max_bytes_per_sec: 400000
# Потолок, до которого адаптивный пейсер может разгоняться (байт/сек).
# Скорость подбирается автоматически по задержке пути - под колено SFU.
# Дефолт 1000000 (~1 МБ/с). Опусти, чтобы жёстко лимитировать. 0 = дефолт.
# max_bytes_per_sec: 1000000
data: data
debug: false
+4 -2
View File
@@ -152,9 +152,11 @@ No extra fields - everything is default.
|-----------|----------|:------------:|
| `vp8.fps` | VP8 stream FPS | `30` |
| `vp8.batch_size` | Frames per tick | `64` |
| `vp8.max_bytes_per_sec` | Wire byte-rate ceiling, bytes/sec | `400000` |
| `vp8.max_bytes_per_sec` | Wire byte-rate probe ceiling, bytes/sec | `1000000` |
`vp8.max_bytes_per_sec` caps the byte-rate the pacer feeds to the video track. The default 400000 (400 KB/s) is a conservative ceiling tuned under the Telemost SFU policer knee: above it the SFU starts throttling forwarding and the stream stalls. Other services have a different knee, so raise this once you have measured your own stable maximum with real runs (bump it up until stalls start, then step back). `0` keeps the default.
The wire rate is paced by a delay-based adaptive controller that auto-discovers each SFU's policer knee at runtime. The policer queues before it drops, so the path RTT inflates ahead of the throughput collapse; the pacer probes the rate up while RTT stays near the path baseline and backs off once it inflates, settling just under the knee on its own. There is nothing to hand-measure per service.
`vp8.max_bytes_per_sec` only caps how high the controller may probe (default 1000000, ~1 MB/s). Lower it to hard-limit a path; `0` keeps the built-in cap. A fresh session starts probing from 400 KB/s (the previous conservative default), so a healthy path only ramps up from the old operating point.
---
+4 -2
View File
@@ -153,9 +153,11 @@ transport. Используй одинаковые traffic-настройки н
|-----------|----------|:------------:|
| `vp8.fps` | FPS VP8 потока | `30` |
| `vp8.batch_size` | Кадров за тик | `64` |
| `vp8.max_bytes_per_sec` | Потолок скорости на проводе, байт/сек | `400000` |
| `vp8.max_bytes_per_sec` | Потолок, до которого пейсер пробует разгоняться, байт/сек | `1000000` |
`vp8.max_bytes_per_sec` ограничивает байтрейт, который пейсер льёт в видеотрек. Дефолт 400000 (400 КБ/с) - консервативный потолок под policer-колено Telemost SFU: выше него SFU начинает резать форвардинг и поток падает. У других сервисов колено другое, поэтому значение можно поднять, измерив свой стабильный максимум реальными запусками (поднимай постепенно, пока не начнутся падения, потом откатись на шаг назад). `0` оставляет дефолт.
Скорость на проводе регулирует адаптивный контроллер на основе задержки: он сам находит policer-колено каждого SFU в рантайме. Policer начинает копить очередь раньше, чем дропать, поэтому RTT пути растёт до того, как обвалится скорость. Пейсер разгоняется, пока RTT держится у базовой линии пути, и сбрасывает скорость, когда RTT раздувается, сам оседая чуть ниже колена. Ничего измерять руками под конкретный сервис не нужно.
`vp8.max_bytes_per_sec` лишь ограничивает, до какого потолка контроллер может разгоняться (дефолт 1000000, ~1 МБ/с). Опусти, чтобы жёстко лимитировать путь; `0` оставляет встроенный потолок. Свежая сессия стартует с 400 КБ/с (прежний консервативный дефолт), так что здоровый путь только разгоняется от старой рабочей точки.
---
+10
View File
@@ -164,6 +164,16 @@ func (r *kcpRuntime) send(msg []byte) error {
return nil
}
// srtt reports KCP's smoothed round-trip estimate in milliseconds, or 0 when
// no sample exists yet. The adaptive pacer reads this to detect the policer
// queueing that precedes a throughput collapse.
func (r *kcpRuntime) srtt() int32 {
if r == nil || r.sess == nil {
return 0
}
return r.sess.GetSRTT()
}
func (r *kcpRuntime) close() {
r.closeOnce.Do(func() {
_ = r.sess.Close()
+28 -8
View File
@@ -9,14 +9,34 @@ import (
const (
defaultFPS = 30
defaultBatchSize = 64
// defaultMaxBytesPerSec paces the wire byte-rate just under the Telemost
// SFU's measured per-slot policer knee. The original 1.2 MiB/s caused the
// SFU to throttle subscriber forwarding after ~42s; 400 KB/s stays well
// within the SFU's comfort zone while still giving useful throughput. This
// is a conservative floor: the policer knee differs per SFU, so operators
// can raise it via Options.MaxBytesPerSec (vp8.max_bytes_per_sec in YAML)
// once they have measured their own service's stable ceiling.
defaultMaxBytesPerSec = 400_000
// defaultMaxBytesPerSec is the upper bound the adaptive pacer may probe up
// to when no explicit ceiling is configured. The pacer (see pacer.go)
// auto-discovers each SFU's real policer knee at runtime from KCP's
// smoothed RTT, so this is just a sanity cap, not a hand-tuned operating
// point. 1 MiB/s matches the throughput target from issue #107 without
// letting a misbehaving path probe unbounded.
defaultMaxBytesPerSec = 1_000_000
// defaultMinProbeRate floors the adaptive rate so a congested path can
// always make forward progress and recover once delay drops. ~120 KB/s
// stays below the worst observed Telemost knee while keeping the control
// plane and keepalives flowing.
defaultMinProbeRate = 120_000
// defaultStartRate is where the pacer begins probing from on a fresh
// session: the old conservative 400 KB/s floor, so behaviour on a healthy
// path only ever ramps up from the previously shipped operating point.
defaultStartRate = 400_000
// Adaptive pacer tuning. Delay marks are derived from the measured path
// baseline plus a fixed slack so a low-RTT LAN and a high-RTT WAN both get
// a sane congestion threshold.
rateProbeSlackMs = 8 // extra ms below which we probe the rate up
rateCongestionSlackMs = 25 // extra ms above which we back the rate off
rateProbeStepBytes = 50_000
rateBackoffNum = 4 // multiplicative decrease: rate *= 4/5
rateBackoffDen = 5
baseRTTDriftDiv = 64 // baseline rises by 1/64 of the gap per sample
)
// Options tunes the vp8channel transport. Zero values fall back to documented defaults.
+81
View File
@@ -0,0 +1,81 @@
package vp8channel
// ratePacer is a delay-based AIMD controller that auto-discovers the wire
// byte-rate a policed carrier (the SFU) tolerates, with no operator tuning.
//
// The carrier policer starts queueing before it starts dropping, so KCP's
// smoothed RTT inflates ahead of the throughput collapse seen in issues #95
// and #107. Each control tick the pacer folds one SRTT sample in: while the
// delay stays near the path baseline it probes the rate up additively, and
// once the delay inflates past the baseline it backs off multiplicatively.
// The rate settles just under the per-SFU policer knee on its own, which is
// the "real run" compromise the fixed 400 KB/s floor only approximated. The
// operator ceiling (vp8.max_bytes_per_sec) just bounds how high it may probe.
type ratePacer struct {
fps int
minRate int
maxRate int
rate int
baseRTT int32
}
func newRatePacer(fps, startRate, minRate, maxRate int) *ratePacer {
if fps <= 0 {
fps = defaultFPS
}
if minRate <= 0 {
minRate = defaultMinProbeRate
}
if maxRate < minRate {
maxRate = minRate
}
return &ratePacer{
fps: fps,
minRate: minRate,
maxRate: maxRate,
rate: max(minRate, min(startRate, maxRate)),
}
}
// perTick is the current per-frame byte budget. The ticker already paces at
// fps, so a per-tick cap bounds the rate without token bookkeeping. Floor at
// one epoch header so keepalives always fit.
func (rp *ratePacer) perTick() int {
pt := rp.rate / rp.fps
if pt < epochHdrLen {
return epochHdrLen
}
return pt
}
// observe folds one smoothed-RTT sample (milliseconds, as KCP reports it) into
// the controller and returns the updated per-tick byte budget. A non-positive
// sample means KCP has no RTT estimate yet, so the rate is held steady.
func (rp *ratePacer) observe(srtt int32) int {
if srtt <= 0 {
return rp.perTick()
}
rp.trackBaseline(srtt)
lowMark := rp.baseRTT + rp.baseRTT/4 + rateProbeSlackMs
highMark := rp.baseRTT + rp.baseRTT/2 + rateCongestionSlackMs
switch {
case srtt >= highMark:
rp.rate = max(rp.minRate, min(rp.rate*rateBackoffNum/rateBackoffDen, rp.maxRate))
case srtt <= lowMark:
rp.rate = max(rp.minRate, min(rp.rate+rateProbeStepBytes, rp.maxRate))
}
return rp.perTick()
}
// trackBaseline anchors baseRTT to the path's uncongested delay: it drops
// instantly to any new minimum and drifts up slowly otherwise, so a genuine
// path change cannot pin the baseline below the real floor forever while a
// transient queueing spike still reads as congestion.
func (rp *ratePacer) trackBaseline(srtt int32) {
if rp.baseRTT == 0 || srtt < rp.baseRTT {
rp.baseRTT = srtt
return
}
rp.baseRTT += (srtt - rp.baseRTT) / baseRTTDriftDiv
}
+110
View File
@@ -0,0 +1,110 @@
package vp8channel
import "testing"
// TestRatePacerProbesUpOnLowDelay verifies the pacer additively raises the
// rate while the path delay stays near the baseline, up to the ceiling.
func TestRatePacerProbesUpOnLowDelay(t *testing.T) {
rp := newRatePacer(30, defaultStartRate, defaultMinProbeRate, defaultMaxBytesPerSec)
// First sample establishes the baseline (20ms). Subsequent equal samples
// read as uncongested, so the rate climbs by rateProbeStepBytes each tick.
rp.observe(20)
start := rp.rate
for range 5 {
rp.observe(20)
}
if rp.rate <= start {
t.Fatalf("rate did not climb on low delay: start=%d now=%d", start, rp.rate)
}
if rp.rate > rp.maxRate {
t.Fatalf("rate %d exceeded ceiling %d", rp.rate, rp.maxRate)
}
}
// TestRatePacerBacksOffOnHighDelay verifies a delay spike past the congestion
// mark triggers a multiplicative decrease.
func TestRatePacerBacksOffOnHighDelay(t *testing.T) {
rp := newRatePacer(30, defaultStartRate, defaultMinProbeRate, defaultMaxBytesPerSec)
rp.observe(20) // baseline 20ms
before := rp.rate
// 20ms baseline -> highMark = 20 + 10 + 25 = 55ms. A 120ms sample is well
// past it and must shrink the rate.
rp.observe(120)
if rp.rate >= before {
t.Fatalf("rate did not back off on high delay: before=%d after=%d", before, rp.rate)
}
want := before * rateBackoffNum / rateBackoffDen
if rp.rate != want {
t.Fatalf("backoff rate = %d, want %d", rp.rate, want)
}
}
// TestRatePacerConvergesToKnee drives a synthetic policer: delay stays low
// until the rate crosses a hidden knee, then inflates. The pacer must settle
// in a band around that knee instead of running away or collapsing.
func TestRatePacerConvergesToKnee(t *testing.T) {
const knee = 600_000
rp := newRatePacer(30, defaultStartRate, defaultMinProbeRate, defaultMaxBytesPerSec)
srtt := func(rate int) int32 {
// Below the knee the path is idle (20ms). Above it the policer queues,
// inflating delay in proportion to the overshoot.
if rate <= knee {
return 20
}
return 20 + int32(min((rate-knee)/10_000, 1_000)) //nolint:gosec // G115: bounded by min to 1000
}
for range 400 {
rp.observe(srtt(rp.rate))
}
// After convergence the rate must sit in a sane band around the knee:
// high enough to beat the old 400 KB/s floor, never pinned at the ceiling.
if rp.rate < defaultStartRate {
t.Fatalf("converged rate %d below start floor %d", rp.rate, defaultStartRate)
}
if rp.rate >= rp.maxRate {
t.Fatalf("rate pinned at ceiling %d, did not detect knee", rp.rate)
}
if rp.rate > knee+4*rateProbeStepBytes {
t.Fatalf("converged rate %d overshoots knee %d", rp.rate, knee)
}
}
// TestRatePacerHoldsWithoutSample verifies a non-positive SRTT (no estimate
// yet) holds the rate steady rather than probing blind.
func TestRatePacerHoldsWithoutSample(t *testing.T) {
rp := newRatePacer(30, defaultStartRate, defaultMinProbeRate, defaultMaxBytesPerSec)
before := rp.rate
if got := rp.observe(0); got != before/rp.fps {
t.Fatalf("perTick on no-sample = %d, want %d", got, before/rp.fps)
}
if rp.rate != before {
t.Fatalf("rate moved without a sample: before=%d after=%d", before, rp.rate)
}
}
// TestRatePacerNeverBelowFloor verifies sustained congestion cannot starve the
// rate below the minimum that keeps the control plane and keepalives flowing.
func TestRatePacerNeverBelowFloor(t *testing.T) {
rp := newRatePacer(30, defaultStartRate, defaultMinProbeRate, defaultMaxBytesPerSec)
rp.observe(10)
for range 100 {
rp.observe(5000) // permanent heavy congestion
}
if rp.rate < rp.minRate {
t.Fatalf("rate %d fell below floor %d", rp.rate, rp.minRate)
}
}
// TestRatePacerPerTickFloor verifies the per-tick budget never drops below one
// epoch header so keepalives always fit even at the rate floor.
func TestRatePacerPerTickFloor(t *testing.T) {
rp := newRatePacer(30, epochHdrLen, epochHdrLen, epochHdrLen)
if got := rp.perTick(); got < epochHdrLen {
t.Fatalf("perTick = %d, want >= %d", got, epochHdrLen)
}
}
+67 -17
View File
@@ -155,7 +155,14 @@ type streamTransport struct {
controlKCPOnce sync.Once
frameInterval time.Duration
batchSize int
perTickBytes int
// Adaptive pacer bounds (bytes/sec). Each writer goroutine builds its own
// ratePacer from these and probes the live wire rate from KCP's SRTT, so
// the operating point auto-tracks each SFU's policer knee. startRate is
// the conservative point a fresh session begins probing from; maxRate is
// the operator ceiling (vp8.max_bytes_per_sec) or the built-in cap.
startRate int
minRate int
maxRate int
// localEpoch is stamped into every outgoing VP8 frame. Explicit
// upper-layer resets rotate it so the peer can reset its KCP state too.
@@ -284,16 +291,13 @@ func newStreamTransport(
if batchSize <= 0 {
batchSize = defaultBatchSize
}
byteRate := opts.MaxBytesPerSec
if byteRate <= 0 {
byteRate = defaultMaxBytesPerSec
}
// Bytes we may emit per frame tick to hold the wire under byteRate. The
// ticker already paces at fps, so a per-tick cap bounds the rate without
// any token bookkeeping. Floor at one epoch header so keepalives fit.
perTickBytes := byteRate / fps
if perTickBytes < epochHdrLen {
perTickBytes = epochHdrLen
// maxRate caps how high the adaptive pacer may probe. An explicit
// vp8.max_bytes_per_sec becomes that ceiling; otherwise the built-in cap
// applies. The pacer auto-discovers the real operating point under it from
// KCP's SRTT, so no value here needs hand-tuning to a specific SFU.
maxRate := opts.MaxBytesPerSec
if maxRate <= 0 {
maxRate = defaultMaxBytesPerSec
}
tr := &streamTransport{
@@ -307,7 +311,9 @@ func newStreamTransport(
writerDone: make(chan struct{}),
frameInterval: time.Second / time.Duration(fps),
batchSize: batchSize,
perTickBytes: perTickBytes,
startRate: defaultStartRate,
minRate: defaultMinProbeRate,
maxRate: maxRate,
bindingToken: bindingToken(cfg.RoomURL),
localEpoch: randomEpoch(),
peers: make(map[uint32]*kcpRuntime),
@@ -335,6 +341,24 @@ func newStreamTransport(
return tr
}
// fps recovers the frame rate from the configured frame interval. The pacer
// divides the per-second rate by this to get a per-tick byte budget.
func (p *streamTransport) fps() int {
if p.frameInterval <= 0 {
return defaultFPS
}
return max(int(time.Second/p.frameInterval), 1)
}
// dataSRTT reports the smoothed RTT (ms) of the single-peer data KCP session,
// or 0 when no session or sample exists yet.
func (p *streamTransport) dataSRTT() int32 {
p.kcpMu.RLock()
rt := p.kcp
p.kcpMu.RUnlock()
return rt.srtt()
}
func (p *streamTransport) Connect(ctx context.Context) error {
connectCtx, cancel := context.WithTimeout(ctx, defaultConnectTimeout)
defer cancel()
@@ -719,14 +743,33 @@ func (p *streamTransport) Features() transport.Features {
type writerState struct {
p *streamTransport
keepaliveEvery int
idleTicks int
forceKeepaliveEvery int
idleTicks int
ticksSinceKeepalive int
// pacer adapts the per-tick byte budget from the live path delay so the
// wire rate tracks the SFU's policer knee without operator tuning.
pacer *ratePacer
// srttFn reports the smoothed RTT (ms) of the KCP session this writer
// feeds, so the pacer can read congestion from the right plane.
srttFn func() int32
// pendingControl holds a control frame that failed WriteSample and must be
// retried on the next tick before consuming more frames.
pendingControl []byte
}
// perTick folds the current path delay into the pacer and returns the byte
// budget for this tick. A nil pacer (defensive) falls back to the start rate.
func (w *writerState) perTick() int {
if w.pacer == nil {
return epochHdrLen
}
var srtt int32
if w.srttFn != nil {
srtt = w.srttFn()
}
return w.pacer.observe(srtt)
}
func (w *writerState) writeSample(data []byte) bool {
return w.p.writeSampleLocked(data)
}
@@ -798,7 +841,7 @@ func (w *writerState) drainControl() bool {
func (w *writerState) drainData() {
select {
case frame := <-w.p.outbound:
sample := w.p.batchSample(frame, w.p.perTickBytes)
sample := w.p.batchSample(frame, w.perTick())
w.idleTicks = 0
_ = w.writeSample(sample)
default:
@@ -821,6 +864,8 @@ func (p *streamTransport) writerLoop() {
p: p,
keepaliveEvery: max(int(keepaliveIdlePeriod/p.frameInterval), 1),
forceKeepaliveEvery: max(int((2*time.Second)/p.frameInterval), 1),
pacer: newRatePacer(p.fps(), p.startRate, p.minRate, p.maxRate),
srttFn: p.dataSRTT,
}
for {
@@ -1348,7 +1393,7 @@ func (p *streamTransport) getOrCreatePeerKCP(epoch uint32) *kcpRuntime {
logger.Infof("vp8channel: peer session created epoch=0x%08x", epoch)
// Pump outbound frames from this peer's queue into the writer.
go p.peerWriterPump(epoch, out)
go p.peerWriterPump(rt, out)
return rt
}
@@ -1361,10 +1406,15 @@ func (p *streamTransport) getOrCreatePeerKCP(epoch uint32) *kcpRuntime {
// overran the SFU's policer, drove burst loss, and collapsed throughput to
// zero within ~20-40s (issue #95). Stops when the channel is closed or the
// transport shuts down.
func (p *streamTransport) peerWriterPump(_ uint32, out chan []byte) {
func (p *streamTransport) peerWriterPump(rt *kcpRuntime, out chan []byte) {
ticker := time.NewTicker(p.frameInterval)
defer ticker.Stop()
// Each downlink peer paces independently: its own ratePacer reads that
// peer's KCP SRTT and tracks its path's policer knee without affecting
// the others or the client->server path.
pacer := newRatePacer(p.fps(), p.startRate, p.minRate, p.maxRate)
// Inject a decodable VP8 keyframe on the same cadence writerLoop uses for
// the client->server path. The server's per-peer bulk path previously
// emitted only opaque KCP data frames, which never form a decodable VP8
@@ -1392,7 +1442,7 @@ func (p *streamTransport) peerWriterPump(_ uint32, out chan []byte) {
if !ok {
return
}
sample := p.batchSampleFrom(out, frame, p.perTickBytes)
sample := p.batchSampleFrom(out, frame, pacer.observe(rt.srtt()))
_ = p.writeSampleLocked(sample)
default:
}
@@ -614,21 +614,19 @@ func TestReorderBufferRestoresOrderAndSurvivesLoss(t *testing.T) {
}
}
// TestMaxBytesPerSecPacing verifies the operator-tunable wire rate cap flows
// into the per-tick byte budget that paces the writer. The default keeps the
// conservative built-in ceiling; an explicit value lets operators dial in
// their own SFU's stable maximum instead of being pinned at the floor. See
// issue #107.
func TestMaxBytesPerSecPacing(t *testing.T) {
// TestMaxBytesPerSecCeiling verifies vp8.max_bytes_per_sec becomes the upper
// bound the adaptive pacer may probe up to, while zero falls back to the
// built-in cap. The pacer auto-discovers the real operating point under that
// ceiling from KCP's SRTT, so the option is a safety bound, not a hand-tuned
// rate. See issue #107.
func TestMaxBytesPerSecCeiling(t *testing.T) {
cases := []struct {
name string
maxBytes int
fps int
wantPerTick int
wantMaxRate int
}{
{name: "default floor", maxBytes: 0, fps: 30, wantPerTick: defaultMaxBytesPerSec / 30},
{name: "explicit raise", maxBytes: 1_000_000, fps: 40, wantPerTick: 1_000_000 / 40},
{name: "clamped to header", maxBytes: 1, fps: 30, wantPerTick: epochHdrLen},
{name: "default cap", maxBytes: 0, wantMaxRate: defaultMaxBytesPerSec},
{name: "explicit ceiling", maxBytes: 2_000_000, wantMaxRate: 2_000_000},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@@ -636,10 +634,13 @@ func TestMaxBytesPerSecPacing(t *testing.T) {
&engineVideoSession{},
nil,
transport.Config{},
Options{FPS: c.fps, BatchSize: 1, MaxBytesPerSec: c.maxBytes},
Options{FPS: 30, BatchSize: 1, MaxBytesPerSec: c.maxBytes},
)
if tr.perTickBytes != c.wantPerTick {
t.Fatalf("perTickBytes = %d, want %d", tr.perTickBytes, c.wantPerTick)
if tr.maxRate != c.wantMaxRate {
t.Fatalf("maxRate = %d, want %d", tr.maxRate, c.wantMaxRate)
}
if tr.startRate != defaultStartRate {
t.Fatalf("startRate = %d, want %d", tr.startRate, defaultStartRate)
}
})
}