Per-server setting ssh_cooldown_base (default 30s, range 5-300) for the
SSH circuit breaker introduced with the event-loop fix. Editable in the
server management modal with a 'leave it alone when healthy' hint, and
shown in the host tuning modal. get_ssh applies edits to the pooled
connection without dropping it.
Long-lived pooled transports were dying between polls on servers behind
stricter NAT/firewall state timeouts, forcing a full SSH handshake on
nearly every poll cycle.
Consecutive connect failures now grow the cooldown 30s -> 60s -> 120s,
capped at 300s, and the first successful connect resets the streak.
connect() itself records failures, so the reconnect-and-retry path in
_run_command_locked feeds the breaker too; fail-fast 'backing off Ns'
behavior is unchanged.
Paramiko raises a bare EOFError/SSHException when the transport goes away
mid-command, and `str(exc)` on those is an empty string. run_command passes
that empty string on as the failure reason, and the managers interpolate it
straight into their messages - about 30 `{err}` sites across managers/ - so a
dropped connection surfaces as "Failed to configure container: " with nothing
after the colon, and in the log as "Command exited with code -1: ".
The SSH layer now falls back to the exception type when the message is empty,
and labels the read failure explicitly: "SSH connection lost while running the
command (EOFError)". Three call sites: the two reconnect paths and the
stdout/stderr read.
Seen in practice where a panel manages a host that rate-limits SSH (ufw's
`limit` on port 22): a burst of connections gets dropped mid-command and every
resulting error message was blank.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pooled SSHManager is shared between request handlers and background
threads (traffic sync, conn monitor). When one side's exec failed, its
reconnect closed the shared client while the other side was mid-command:
'NoneType open_session', EOFError in open_sftp, a burst of 500s, then
30s of fast-fail from the connect cooldown.
Add an RLock around exec_command and all SFTP operations (RLock because
run_command retries by re-entering itself). Lock ordering is
exec -> conn, connect/disconnect never take the exec lock, so no
deadlock. Failed execs now serialize the reconnect too: other threads
simply wait and then run on the fresh transport.
- per-manager lock around connect/disconnect: UI handler and background
monitor can no longer race a half-built transport
- transport keepalive 30s: NAT/firewalls won't silently drop idle tunnels
- one reconnect+retry when exec fails on a dead-but-'active' transport
instead of surfacing a spurious error to the UI
- SSHManager.ensure_connected(): keep the transport alive, reconnect only
when it died; run_command auto-heals the connection
- app.py: pool SSHManager per (host, port, username) instead of a fresh
TCP+SSH handshake for every API call - the main cause of UI timeouts on
high-latency servers (each 6s refresh paid a 1-2s handshake)
- awg_manager: 10s TTL cache for config path resolution and server config
reads (was: the same awg0.conf cat'ed 3-4x per refresh); caches are
invalidated on save_server_config
Add SOCKS5 support
Add AdGuard Home integration
Add node status and latency tracking
Add drag-and-drop server management
Upgrade Xray from v1.8 to v26.3.27 with backward compatibility
Fix Telemt dependency installation
Extend Telegram bot to support all protocols
Improve server cleanup process
Enhance API structure (system routes moved to a separate group)
Add JWT token generation for external integrations
Максимов Владимир Сергеевич