Files
siyuan/kernel/api/network.go
T
Yingyi / 颖逸 e564ce7b1f ✨ Support kernel plugin system (#17487)
* ♻️ Add/update indirect Go dependencies in kernel

Update kernel/go.mod and kernel/go.sum to add multiple indirect modules and checksum entries. Notable additions include github.com/fastschema/qjs, github.com/filecoin-project/go-jsonrpc, github.com/ipfs/go-log/v2, go.opencensus.io, go.uber.org/{atomic,multierr,zap}, golang.org/x/xerrors and github.com/golang/groupcache among many transitive entries. Changes ensure transitive dependencies are pinned and go.sum checksums are present (likely produced by `go mod tidy`) to make builds reproducible.

* refactor: export bazaar.GetCurrentBackend for kernel plugin platform matching

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* build: promote qjs to direct dependency for kernel plugin system

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(plugin): add KernelPlugin struct with QJS runtime lifecycle and state machine

Introduces plugin/plugin.go with KernelPlugin owning an isolated QuickJS
runtime, a mutex-serialized call path, RPC method registration/dispatch,
Promise awaiting, JSON round-trip result conversion, and WebSocket tracking.
Adds sandbox_stub.go as a temporary no-op stub for injectSandboxGlobals.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(plugin): add PluginManager singleton for kernel plugin discovery and lifecycle

* feat(plugin): add sandbox injection scaffold with siyuan.log

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(plugin): implement siyuan.storage CRUD scoped to petal storage directory

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(plugin): implement siyuan.fetch with browser-like Response interface

* feat(plugin): implement siyuan.socket with browser-compatible WebSocket API

- Add sync import for mutex-protected WebSocket connection tracking
- Implement __siyuan_socket Go function that creates browser-compatible WebSocket objects
- Support send() method with queueing for messages sent before connection opens
- Support close() method for closing the WebSocket connection
- Track connection state via readyState property (0=CONNECTING, 1=OPEN, 3=CLOSED)
- Connect to kernel WebSocket endpoint with automatic auth token injection
- Run WebSocket I/O in background goroutine with proper cleanup
- Wire up siyuan.socket JS API

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(plugin): implement siyuan.rpc.register for JSON-RPC method registration

* feat(plugin): add JSON-RPC 2.0 handler for kernel plugin method dispatch

* feat(plugin): register /api/plugin/rpc/:name and /ws/plugin/rpc/:name routes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(plugin): wire kernel plugin manager start/stop into main lifecycle

* feat(plugin): hook SetPetalEnabled to start/stop kernel plugins on enable/disable

* test(plugin): add unit tests for kernel plugin state machine and eligibility

* test(plugin): add comprehensive unit tests for manager, sandbox, and RPC handlers

* refactor(plugin): Export IsTargetSupported and update usages

Rename isTargetSupported to exported IsTargetSupported and adjust its comment. Replace local calls with bazaar.IsTargetSupported in kernel/bazaar and kernel/plugin/manager, removing the duplicated isKernelEligible helper. Update tests to import bazaar, call the new function, and change expectations to reflect that nil/empty kernel slices are treated as supported (i.e. supported on all platforms).

* refactor(plugin): initialize PluginManager in main and update related usages

* refactor(plugin): update JWT handling and plugin initialization for kernel plugins

* refactor(plugin): enhance plugin initialization and improve sandbox global injections

* refactor(kernel-plugin): Refactor plugin RPC registration and sandbox integration

- Removed deprecated tests and refactored existing tests for clarity and efficiency.
- Updated RPC method registration to use `bind` and `unbind` methods for better clarity.
- Enhanced the `injectSandboxGlobals` function to include additional properties for the plugin.
- Improved error handling in RPC methods and ensured proper state management for plugins.
- Added benchmarks for map to JS conversion performance.
- Cleaned up unused imports and organized code structure for better readability.

* refactor(plugin): enhance concurrency handling and improve WebSocket integration

* refactor(kernel-plugin): enhance RPC method handling and improve function registration

* feat(kernel-plugin): add RPC method info retrieval and enhance plugin management

* refactor(plugin): add plugin management endpoints and enhance plugin info retrieval

* refactor(kernel-plugin): enhance RPC method handling and improve plugin info retrieval

* refactor(kernel-plugin): improve error handling and response structures in RPC methods

* refactor(kernel-plugin): improve error handling in RPC methods and enhance WebSocket closure management

* fix(kernel-plugin): initialize sockets and socketMus maps in NewKernelPlugin

* feat(kernel-plugin): add wsWrite helper and fix PushNotification omitempty

Add wsWrite method on KernelPlugin that acquires the per-connection write
mutex before sending a text frame, returning nil for untracked connections.
Fix PushNotification's Params field to use omitempty for JSON-RPC 2.0 §4.2
compliance. Add rpc_test.go with newTestWsPair helper and tests for wsWrite.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(kernel-plugin): add BroadcastNotification and per-connection write mutex

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(kernel-plugin): expose siyuan.rpc.broadcast in plugin sandbox

Add rpc.broadcast(method, params) binding in injectRpc so JS plugins
can push JSON-RPC 2.0 notifications to all connected server clients.
Fix deadlock by introducing a dedicated socketsMu RWMutex for the
sockets map, decoupling socket tracking from the main plugin mutex
that is held during Start()/Eval().

* fix(kernel-plugin): double-unlock in send handler and document PushNotification write-safety

Remove spurious mu.Unlock() inside the nil-conn branch of injectSocket's
CONNECTING-state send handler; the outer unconditional unlock is sufficient,
so the inner one causes a panic under concurrent load.

Document that PushNotification bypasses per-connection write serialization
and must not be called concurrently with BroadcastNotification/wsWrite on
the same connection without external locking.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style(kernel-plugin): align struct field declarations in KernelPlugin

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(kernel-plugin): omit params field from JsonRpcRequest when nil (JSON-RPC 2.0 §4.1)

Per spec, params MAY be omitted; add omitempty so marshaled requests
with no parameters do not emit "params":null.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(kernel-plugin): change JsonRpcRequest.Params to *json.RawMessage

A pointer correctly models the three-way distinction:
- nil      → params key absent (omitted from marshal output via omitempty)
- non-nil → params present (null, array, or object)

The previous []byte omitempty omitted the key only for nil/empty slices
and could not distinguish absent from explicit null on the wire.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(kernel-plugin): unify method naming conventions and improve JSON-RPC request handling

* fix(kernel-plugin): improve WebSocket message handling and ensure thread safety with mutexes

* fix(kernel-plugin): enhance WebSocket handling and improve error management in storage methods

* fix(kernel-plugin): rename JsonRpcRequestRaw to JsonRpcInboundRequest and update related methods

* fix(kernel-plugin): improve plugin management and error handling in kernel plugin methods

* fix(kernel-plugin): rename kernel field to kernels and update related references

* feat(kernel-plugin): implement logging and improve concurrency handling in plugin manager and storage methods

* feat(kernel-plugin): enhance RPC parameter handling and add JSON array parsing support

* refactor(kernel-plugin): refactor RPC handling and improve logging functionality

* refactor(kernel-plugin): streamline loggerWrapper function and improve error handling in injectFetch

* refactor(kernel-plugin): optimize injectFetch function and enhance error handling

* feat(kernel-plugin): add onLoaded hook and enhance plugin lifecycle management

* feat(kernel-plugin): add ObjectFreeze and ObjectSeal functions to enhance API security

* feat(kernel-plugin): add InitJwtKey function to generate JWT signing key

* refactor(kernel-plugin): enhance error handling and logging in plugin lifecycle methods

* feat(kernel-plugin): improve WebSocket error handling and add concurrency support in BroadcastNotification

* feat(kernel-plugin): enhance error handling in storage and fetch methods with panic recovery

* feat(kernel-plugin): enhance PluginManager concurrency and error handling with sync.Map and atomic operations

* feat(kernel-plugin): refactor PluginState to use atomic operations for improved concurrency

* feat(kernel-plugin): add PluginStateLoaded and update state management in plugin lifecycle

* refactor(kernel-plugin): update logging level in loadPetals and refactor loggerWrapper return values

* feat(kernel-plugin): simplify invokeHook and enhance error handling in Object methods

* feat(kernel-plugin): remove obsolete test files for plugin functionality

* refactor(kernel-plugin): implement loggerWrapper and rpcParamsToJsValue functions for improved logging and RPC parameter handling

* feat(kernel-plugin): introduce Worker for serializing plugin tasks and enhance context management

* refactor(worker): enhance task execution with callback support and graceful shutdown

- Introduced a callback mechanism in the Task struct to handle results and errors.
- Updated the Run method to accept a callback, allowing immediate handling of task results.
- Added a RunSync method for synchronous task execution with result retrieval.
- Implemented atomic closure state management to prevent task submission after closure.
- Enhanced the Close method to ensure graceful shutdown and wait for the worker to finish processing.

* feat(kernel-plugin): refactor storage and RPC methods to use PromiseRun for better error handling

* feat(kernel-plugin): enhance plugin event handling with lifecycle and RPC event subscriptions

* refactor(kernel-plugin): replace PromiseRun with worker.Run for improved error handling in event and storage methods

* chore(kernel-plugin): add goja dependency, drop qjs

* chore(kernel-plugin): delete KernelPluginLogger (qjs stdout/stderr only)

* refactor(kernel-plugin): replace qjs runtime with goja in plugin.go

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(kernel-plugin): add sandbox utility tests (pre-rewrite)

* refactor(kernel-plugin): rewrite sandbox utility functions for goja

Replace goValueToJsValue, getJsContextValue, dispatchEvent with goja
implementations; add convertJsonNumbers helper; stub ObjectFreeze and
ObjectSeal as no-ops; delete dead qjs-only helpers (invokeRpcMethod,
PromiseAwait, rpcParamsToJsValue, parseJsonArrayStringToJsValueArray,
parseJsonStringToJsValue, loggerWrapper, ObjectSetDataMethods).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(kernel-plugin): rewrite sandbox.go inject functions for goja

Replace all qjs-based inject functions (injectGlobalContext, injectPlugin,
injectLogger, injectEvent, injectStorage, injectFetch, injectSocket, injectRpc)
with goja equivalents. Add ObjectSetDataMethods and loggerWrapper helpers.
Remove all remaining qjs dead code; ObjectFreeze/ObjectSeal now call
Object.freeze/seal via goja AssertFunction.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(kernel-plugin): add plugin lifecycle and RPC integration tests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore(kernel-plugin): go mod tidy after qjs removal

Remove fastschema/qjs from go.mod and go.sum, add go-sourcemap as
indirect (transitive dep of dop251/goja), mark go-sourcemap indirect.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix(kernel-plugin): fix invokeHook early-return on subscribe failure, safe await extraction, and goja value cross-goroutine access in socket methods

* refactor(kernel-plugin): replace goValueToJsValue with goValueToJsValueSafely in sandbox functions and tests

* feat(plugin): enhance plugin management and error handling

- Added GetLoadedPlugin method to retrieve loaded plugin info by name.
- Introduced file path for kernel.js in KernelPlugin struct.
- Updated Eval method to use the new file path for script execution.
- Improved error handling in injectGlobalContext and other injection functions using recover.
- Refactored task execution in Worker to use clearer types for task executors and callbacks.
- Enhanced storage methods to ensure proper error handling and logging.
- Updated loggerWrapper to handle errors more gracefully.
- Ensured consistent use of error handling patterns across various plugin methods.

* refactor(worker): enhance task execution with goja runtime integration

- Updated TaskExecutor and TaskCallback signatures to accept *goja.Runtime.
- Modified Worker to start processing tasks with an event loop.
- Improved error handling in task execution to catch panics from both executor and callback.
- Renamed Close method to Stop for clarity on worker shutdown behavior.

* refactor(kernel-plugin): streamline worker implementation and update context handling in plugin methods

* refactor(kernel-plugin): update event handler to use byte slices and improve event dispatching

* refactor(worker): simplify RunSync method by removing unnecessary select statement

* refactor(kernel-plugin): enhance plugin lifecycle management and improve RPC method binding

* refactor(kernel-plugin): improve error logging in data methods for better debugging

* refactor(kernel-plugin): add version field to plugin data structures and update related methods

* refactor(kernel-plugin): replace JsonRpcInboundRequest with JsonRpcRequest and update related methods

* refactor(kernel-plugin): enhance plugin lifecycle hooks and improve RPC method invocation

* feat(kernel-plugin): improve error handling and response processing in fetch and socket methods

* refactor(kernel-plugin): update invokeFunction to handle promise results correctly

* refactor(kernel-plugin): streamline event handling and remove unused JSON marshaling functions

* refactor(kernel-plugin): improve error handling in start method and add event publishing for lifecycle states

* refactor(kernel-plugin): move logging to separate function and execute in goroutines for improved performance

* feat(kernel-plugin): add unique ID generation for start and stop events

* refactor(kernel-plugin): enhance error handling and concurrency in storage operations

Co-authored-by: Copilot <copilot@github.com>

* fix(kernel-plugin): remove unexpected resolve in fetch function

* feat(kernel-plugin): enhance JSON-RPC request handling with optional parameters and improved error reporting

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): rename await to async in dispatchEvent function for clarity

Co-authored-by: Copilot <copilot@github.com>

* fix(kernel-plugin): improve error handling in RPC method execution and hook invocation

* feat(kernel-plugin): implement custom JSON marshaling for JsonRpcRequest to handle optional parameters

* feat(kernel-plugin): add error codes for plugin state and improve error handling in RPC responses

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): clean up context usage and improve error logging for RPC methods

* feat(kernel-plugin): add buffer method to object for asynchronous data processing

* fix(kernel-plugin): Fixed the problem of blocking when plug-in life cycle function is not bound

Co-authored-by: Copilot <copilot@github.com>

* feat(kernel-plugin): implement public and private web server handlers and enhance request handling

Co-authored-by: Copilot <copilot@github.com>

* feat(kernel-plugin): enhance server request handling and introduce server handler invocation

Co-authored-by: Copilot <copilot@github.com>

* feat(kernel-plugin): enhance response handling and add jsValueToBytes conversion utility

Co-authored-by: Copilot <copilot@github.com>

* feat(kernel-plugin): comment out public web server route in router

* feat(kernel-plugin): add WebSocket and EventSource proxy handlers and update sandbox integration

Co-authored-by: Copilot <copilot@github.com>

* feat(kernel-plugin): implement HTTP proxy handler with response header forwarding

* refactor(kernel-plugin): refactor siyuan.client.* methods

* feat(kernel-plugin): add support for EventSource with SSE handling and response header forwarding

Co-authored-by: Copilot <copilot@github.com>

* feat(kernel-plugin): add SSE support using r3labs/sse library for EventSource handling

* feat(kernel-plugin): enhance SSE client with onclose event handling

Co-authored-by: Copilot <copilot@github.com>

* feat(kernel-plugin): implement SSE event handling and error management in server-sent events

* feat(kernel-plugin): refactor SSE handling and introduce request handler utility functions

Co-authored-by: Copilot <copilot@github.com>

* feat(kernel-plugin): enhance WebSocket message handling with buffered amount tracking and cleanup

Co-authored-by: Copilot <copilot@github.com>

* perf(kernel-plugin): improve WebSocket message handling with channel-based message sending and error management

Co-Authored-By: Copilot <copilot@github.com>

* refactor(kernel-plugin): remove invokeServerHandler

Co-Authored-By: Copilot <copilot@github.com>

* feat(kernel-plugin): implement WebSocket message handling with improved structure and error management

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): Refactor code structure for improved readability and maintainability

* refactor(kernel-plugin): streamline HTTP client creation and enhance event source state management

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): enhance WebSocket and SSE handling with improved closure management and error handling

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): optimize WebSocket handling by restructuring state management and improving closure logic

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): simplify header setting and improve null checks in WebSocket and SSE handling

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): update WebSocket request handling to improve error management and consistency

* refactor(kernel-plugin): improve WebSocket error handling by adding close message management

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): Refactor WebSocket handling to use gws library

- Replaced gorilla/websocket with lxzan/gws for WebSocket connections.
- Introduced gwsEventHandler to manage WebSocket events with customizable callbacks.
- Updated KernelPlugin to track gws connections and handle message broadcasting.
- Refactored RPC WebSocket handling to accommodate new gws structure.
- Simplified message sending and connection management logic.
- Added utility function to check for undefined JavaScript values.

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): integrate gws library for improved WebSocket handling and error management

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): remove unnecessary error handling in WebSocket request processing

* refactor(kernel-plugin): enhance error logging in WebSocket message handling

Co-Authored-By: Copilot <copilot@github.com>

* refactor(kernel-plugin): replace gwsEventHandler with WsEventHandler and improve WebSocket management

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): integrate chanx for improved event handling in SSE

* refactor(kernel-plugin): update handleHttpRequest signature to include gin.Context for improved request handling

Co-authored-by: Copilot <copilot@github.com>

* refactor(kernel-plugin): optimize WebSocket connection management with context and sync mechanisms

* refactor(kernel-plugin): improve error handling and context management in WebSocket and HTTP request handling

* refactor(kernel-plugin): enhance WebSocket management with context handling and improved error reporting

* fix(kernel-plugin): streamline header export and enhance error handling in injectClient function

Co-authored-by: Copilot <copilot@github.com>

* perf(kernel-plugin): enhance httpProxy and esProxy functions with improved error handling and content management

Co-authored-by: Copilot <copilot@github.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <copilot@github.com>
2026-05-09 11:26:37 +08:00

627 lines
17 KiB
Go

// SiYuan - Refactor your thinking
// Copyright (c) 2020-present, b3log.org
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <https://www.gnu.org/licenses/>.
package api
import (
"encoding/base32"
"encoding/base64"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net"
"net/http"
"net/textproto"
"net/url"
"strings"
"syscall"
"time"
"github.com/88250/gulu"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/imroc/req/v3"
"github.com/siyuan-note/logging"
"github.com/siyuan-note/siyuan/kernel/util"
)
type File struct {
Filename string
Header textproto.MIMEHeader
Size int64
Content string
}
type MultipartForm struct {
Value map[string][]string
File map[string][]File
}
func echo(c *gin.Context) {
ret := gulu.Ret.NewResult()
defer c.JSON(http.StatusOK, ret)
var (
password string
multipartForm *MultipartForm
rawData any
)
if form, err := c.MultipartForm(); err != nil || nil == form {
multipartForm = nil
} else {
multipartForm = &MultipartForm{
Value: form.Value,
File: map[string][]File{},
}
for k, handlers := range form.File {
files := make([]File, len(handlers))
multipartForm.File[k] = files
for i, handler := range handlers {
files[i].Filename = handler.Filename
files[i].Header = handler.Header
files[i].Size = handler.Size
if file, err := handler.Open(); err != nil {
logging.LogWarnf("echo open form [%s] file [%s] error: %s", k, handler.Filename, err.Error())
} else {
content := make([]byte, handler.Size)
if n, err := file.Read(content); err != nil {
logging.LogWarnf("echo read form [%s] file [%s] error: %s", k, handler.Filename, err.Error())
} else {
files[i].Content = base64.StdEncoding.EncodeToString(content[:n])
}
}
}
}
}
if data, err := c.GetRawData(); err == nil {
rawData = base64.StdEncoding.EncodeToString(data)
} else {
logging.LogWarnf("echo get raw data error: %s", err.Error())
rawData = nil
}
username, password, ok := c.Request.BasicAuth()
ret.Data = map[string]any{
"Context": map[string]any{
"Params": c.Params,
"HandlerNames": c.HandlerNames(),
"FullPath": c.FullPath(),
"ClientIP": c.ClientIP(),
"RemoteIP": c.RemoteIP(),
"ContentType": c.ContentType(),
"IsWebsocket": c.IsWebsocket(),
"RawData": rawData,
},
"Request": map[string]any{
"Method": c.Request.Method,
"URL": c.Request.URL,
"Proto": c.Request.Proto,
"ProtoMajor": c.Request.ProtoMajor,
"ProtoMinor": c.Request.ProtoMinor,
"Header": c.Request.Header,
"ContentLength": c.Request.ContentLength,
"TransferEncoding": c.Request.TransferEncoding,
"Close": c.Request.Close,
"Host": c.Request.Host,
"Form": c.Request.Form,
"PostForm": c.Request.PostForm,
"MultipartForm": multipartForm,
"Trailer": c.Request.Trailer,
"RemoteAddr": c.Request.RemoteAddr,
"TLS": c.Request.TLS,
"UserAgent": c.Request.UserAgent(),
"Cookies": c.Request.Cookies(),
"Referer": c.Request.Referer(),
},
"URL": map[string]any{
"EscapedPath": c.Request.URL.EscapedPath(),
"EscapedFragment": c.Request.URL.EscapedFragment(),
"String": c.Request.URL.String(),
"Redacted": c.Request.URL.Redacted(),
"IsAbs": c.Request.URL.IsAbs(),
"Query": c.Request.URL.Query(),
"RequestURI": c.Request.URL.RequestURI(),
"Hostname": c.Request.URL.Hostname(),
"Port": c.Request.URL.Port(),
},
"User": map[string]any{
"Exists": ok,
"Username": username,
"Password": password,
},
}
}
func forwardProxy(c *gin.Context) {
ret := gulu.Ret.NewResult()
defer c.JSON(http.StatusOK, ret)
arg, ok := util.JsonArg(c, ret)
if !ok {
return
}
var destURL string
if !util.ParseJsonArgs(arg, ret, util.BindJsonArg("url", &destURL, true, true)) {
return
}
u, e := url.ParseRequestURI(destURL)
if nil != e {
ret.Code = -1
ret.Msg = "invalid [url]"
return
}
if u.Scheme != "http" && u.Scheme != "https" {
ret.Code = -1
ret.Msg = "only http/https is allowed"
return
}
method := "POST"
if methodArg := arg["method"]; nil != methodArg {
method = strings.ToUpper(methodArg.(string))
}
timeout := 7000
if timeoutArg := arg["timeout"]; nil != timeoutArg {
timeout = int(timeoutArg.(float64))
if 1 > timeout {
timeout = 7000
}
}
client := getSafeClient(time.Duration(timeout) * time.Millisecond)
request := client.R()
if headers, ok := arg["headers"].([]any); ok {
for _, pair := range headers {
if m, ok := pair.(map[string]any); ok {
for k, v := range m {
request.SetHeader(k, fmt.Sprintf("%v", v))
}
}
}
}
contentType := "application/json"
if contentTypeArg := arg["contentType"]; nil != contentTypeArg {
contentType = contentTypeArg.(string)
}
request.SetHeader("Content-Type", contentType)
payloadEncoding := "json"
if payloadEncodingArg := arg["payloadEncoding"]; nil != payloadEncodingArg {
payloadEncoding = payloadEncodingArg.(string)
}
switch payloadEncoding {
case "base64":
fallthrough
case "base64-std":
if payload, err := base64.StdEncoding.DecodeString(arg["payload"].(string)); err != nil {
ret.Code = -2
ret.Msg = "decode base64-std payload failed: " + err.Error()
return
} else {
request.SetBody(payload)
}
case "base64-url":
if payload, err := base64.URLEncoding.DecodeString(arg["payload"].(string)); err != nil {
ret.Code = -2
ret.Msg = "decode base64-url payload failed: " + err.Error()
return
} else {
request.SetBody(payload)
}
case "base32":
fallthrough
case "base32-std":
if payload, err := base32.StdEncoding.DecodeString(arg["payload"].(string)); err != nil {
ret.Code = -2
ret.Msg = "decode base32-std payload failed: " + err.Error()
return
} else {
request.SetBody(payload)
}
case "base32-hex":
if payload, err := base32.HexEncoding.DecodeString(arg["payload"].(string)); err != nil {
ret.Code = -2
ret.Msg = "decode base32-hex payload failed: " + err.Error()
return
} else {
request.SetBody(payload)
}
case "hex":
if payload, err := hex.DecodeString(arg["payload"].(string)); err != nil {
ret.Code = -2
ret.Msg = "decode hex payload failed: " + err.Error()
return
} else {
request.SetBody(payload)
}
case "text":
default:
request.SetBody(arg["payload"])
}
started := time.Now()
resp, err := request.Send(method, destURL)
if err != nil {
ret.Code = -1
ret.Msg = "forward request failed: " + err.Error()
return
}
bodyData, err := io.ReadAll(resp.Body)
if err != nil {
ret.Code = -1
ret.Msg = "read response body failed: " + err.Error()
return
}
elapsed := time.Since(started)
responseEncoding := "text"
if responseEncodingArg := arg["responseEncoding"]; nil != responseEncodingArg {
responseEncoding = responseEncodingArg.(string)
}
body := ""
switch responseEncoding {
case "base64":
fallthrough
case "base64-std":
body = base64.StdEncoding.EncodeToString(bodyData)
case "base64-url":
body = base64.URLEncoding.EncodeToString(bodyData)
case "base32":
fallthrough
case "base32-std":
body = base32.StdEncoding.EncodeToString(bodyData)
case "base32-hex":
body = base32.HexEncoding.EncodeToString(bodyData)
case "hex":
body = hex.EncodeToString(bodyData)
case "text":
fallthrough
default:
responseEncoding = "text"
body = string(bodyData)
}
data := map[string]any{
"url": destURL,
"status": resp.StatusCode,
"contentType": resp.GetHeader("content-type"),
"body": body,
"bodyEncoding": responseEncoding,
"headers": resp.Header,
"elapsed": elapsed.Milliseconds(),
}
ret.Data = data
//shortBody := ""
//if 64 > len(body) {
// shortBody = body
//} else {
// shortBody = body[:64]
//}
//
//logging.LogInfof("elapsed [%.1fs], length [%d], request [url=%s, headers=%s, content-type=%s, body=%s], status [%d], body [%s]",
// elapsed.Seconds(), len(bodyData), data["url"], headers, contentType, arg["payload"], data["status"], shortBody)
}
// ssrfSafeDialer returns a net.Dialer whose Control hook blocks private IPs.
func ssrfSafeDialer(timeout time.Duration) *net.Dialer {
return &net.Dialer{
Timeout: timeout,
Control: func(network, address string, _ syscall.RawConn) error {
host, _, err := net.SplitHostPort(address)
if err != nil {
return err
}
if ip := net.ParseIP(host); ip != nil && isPrivateIP(ip) {
return fmt.Errorf("ip address [%s] is prohibited", host)
}
return nil
},
}
}
// 校验 IP 是否为私有内网地址
func isPrivateIP(ip net.IP) bool {
return ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsPrivate() || ip.IsUnspecified()
}
// 创建安全的 HTTP Client,防止 SSRF 和 DNS 重绑定
func getSafeClient(timeout time.Duration) *req.Client {
dialer := ssrfSafeDialer(timeout)
client := req.C()
client.SetTimeout(timeout)
client.SetDial(dialer.DialContext)
client.SetRedirectPolicy(req.MaxRedirectPolicy(3))
return client
}
// parseForwardProxyParams decodes the `u` and `h` query parameters.
//
// Query params:
// - `u`: RawURLEncoding base64 of the target URL string.
// - `h`: RawURLEncoding base64 of a JSON object map[string][]string.
func parseForwardProxyParams(c *gin.Context) (parsedURL *url.URL, headers *http.Header, err error) {
uParam := c.Query("u")
if uParam == "" {
err = fmt.Errorf("missing query param [u]")
return
}
uBytes, decErr := base64.RawURLEncoding.DecodeString(uParam)
if decErr != nil {
err = fmt.Errorf("decode [u] failed: %s", decErr.Error())
return
}
parsedURL, err = url.ParseRequestURI(string(uBytes))
if err != nil {
err = fmt.Errorf("parse [u] failed: %s", err.Error())
return
}
h := http.Header{}
headers = &h
hParam := c.Query("h")
if hParam == "" {
return
}
hBytes, decErr := base64.RawURLEncoding.DecodeString(hParam)
if decErr != nil {
err = fmt.Errorf("decode [h] failed: %s", decErr.Error())
return
}
var record map[string][]string
if jsonErr := json.Unmarshal(hBytes, &record); jsonErr != nil {
err = fmt.Errorf("parse [h] failed: %s", jsonErr.Error())
return
}
for k, vs := range record {
for _, v := range vs {
h.Add(k, v)
}
}
return
}
// forwardResponseHeaders copies src headers into dst with a "Siyuan-Proxy-" prefix on each key.
func forwardResponseHeaders(dst http.Header, src http.Header) {
for k, vs := range src {
for _, v := range vs {
dst.Add("Siyuan-Proxy-"+k, v)
}
}
}
// httpProxy proxies an HTTP request to a remote HTTP endpoint.
//
// Query params:
// - u: RawURLEncoding base64 of the target http/https URL
// - h: RawURLEncoding base64 of JSON map[string][]string forwarded as request headers
//
// The request method and body are taken from the incoming request.
// Target response headers are forwarded with a "Siyuan-Proxy-" prefix.
func httpProxy(c *gin.Context) {
targetURL, targetHeaders, err := parseForwardProxyParams(c)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"code": -1, "msg": err.Error()})
return
}
if targetURL.Scheme != "http" && targetURL.Scheme != "https" {
c.JSON(http.StatusBadRequest, gin.H{"code": -1, "msg": "only http/https is allowed"})
return
}
transport := &http.Transport{
DialContext: ssrfSafeDialer(30 * time.Second).DialContext,
}
httpClient := &http.Client{Transport: transport}
proxyReq, reqErr := http.NewRequestWithContext(c.Request.Context(), c.Request.Method, targetURL.String(), c.Request.Body)
if reqErr != nil {
c.JSON(http.StatusBadRequest, gin.H{"code": -1, "msg": "create request failed: " + reqErr.Error()})
return
}
proxyReq.ContentLength = c.Request.ContentLength
proxyReq.Header.Set("Content-Type", c.ContentType())
for k, vs := range *targetHeaders {
for _, v := range vs {
proxyReq.Header.Add(k, v)
}
}
resp, respErr := httpClient.Do(proxyReq)
if respErr != nil {
c.JSON(http.StatusBadGateway, gin.H{"code": -1, "msg": "connect target failed: " + respErr.Error()})
return
}
defer resp.Body.Close()
forwardResponseHeaders(c.Writer.Header(), resp.Header)
c.Writer.WriteHeader(resp.StatusCode)
if _, err := io.Copy(c.Writer, resp.Body); err != nil {
logging.LogWarnf("http proxy copy response failed: %s", err.Error())
}
}
// wsProxy proxies a WebSocket connection to a remote WebSocket endpoint.
//
// Query params:
// - u: RawURLEncoding base64 of the target ws/wss URL
// - h: RawURLEncoding base64 of JSON map[string][]string forwarded as handshake headers
//
// Target response headers are forwarded with a "Siyuan-Proxy-" prefix.
func wsProxy(c *gin.Context) {
targetURL, targetHeaders, err := parseForwardProxyParams(c)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"code": -1, "msg": err.Error()})
return
}
if targetURL.Scheme != "ws" && targetURL.Scheme != "wss" {
c.JSON(http.StatusBadRequest, gin.H{"code": -1, "msg": "only ws/wss is allowed"})
return
}
wsDialer := &websocket.Dialer{
NetDialContext: ssrfSafeDialer(30 * time.Second).DialContext,
HandshakeTimeout: 30 * time.Second,
}
targetConn, targetResp, dialErr := wsDialer.DialContext(c.Request.Context(), targetURL.String(), *targetHeaders)
if dialErr != nil {
c.JSON(http.StatusBadGateway, gin.H{"code": -1, "msg": "dial target failed: " + dialErr.Error()})
return
}
defer targetConn.Close()
upgradeHeaders := http.Header{}
if targetResp != nil {
forwardResponseHeaders(upgradeHeaders, targetResp.Header)
}
upgrader := websocket.Upgrader{
CheckOrigin: func(r *http.Request) bool { return true },
}
clientConn, upgradeErr := upgrader.Upgrade(c.Writer, c.Request, upgradeHeaders)
if upgradeErr != nil {
logging.LogErrorf("ws forward proxy upgrade failed: %s", upgradeErr.Error())
return
}
defer clientConn.Close()
errChan := make(chan error, 2)
go func() {
for {
msgType, msg, readErr := targetConn.ReadMessage()
if readErr != nil {
if closeError, ok := readErr.(*websocket.CloseError); ok {
clientConn.WriteMessage(
websocket.CloseMessage,
websocket.FormatCloseMessage(
closeError.Code,
closeError.Text,
),
)
}
errChan <- readErr
return
}
if writeErr := clientConn.WriteMessage(msgType, msg); writeErr != nil {
errChan <- writeErr
return
}
}
}()
go func() {
for {
msgType, msg, readErr := clientConn.ReadMessage()
if readErr != nil {
if closeError, ok := readErr.(*websocket.CloseError); ok {
targetConn.WriteMessage(
websocket.CloseMessage,
websocket.FormatCloseMessage(
closeError.Code,
closeError.Text,
),
)
}
errChan <- readErr
return
}
if writeErr := targetConn.WriteMessage(msgType, msg); writeErr != nil {
errChan <- writeErr
return
}
}
}()
<-errChan
}
// esProxy proxies an EventSource (SSE) stream from a remote HTTP endpoint.
//
// Query params:
// - u: RawURLEncoding base64 of the target http/https URL
// - h: RawURLEncoding base64 of JSON map[string][]string forwarded as request headers
//
// Target response headers are forwarded with a "Siyuan-Proxy-" prefix.
func esProxy(c *gin.Context) {
targetURL, targetHeaders, err := parseForwardProxyParams(c)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"code": -1, "msg": err.Error()})
return
}
if targetURL.Scheme != "http" && targetURL.Scheme != "https" {
c.JSON(http.StatusBadRequest, gin.H{"code": -1, "msg": "only http/https is allowed"})
return
}
transport := &http.Transport{
DialContext: ssrfSafeDialer(30 * time.Second).DialContext,
}
httpClient := &http.Client{Transport: transport}
proxyReq, reqErr := http.NewRequestWithContext(c.Request.Context(), http.MethodGet, targetURL.String(), nil)
if reqErr != nil {
c.JSON(http.StatusBadRequest, gin.H{"code": -1, "msg": "create request failed: " + reqErr.Error()})
return
}
for k, vs := range *targetHeaders {
for _, v := range vs {
proxyReq.Header.Add(k, v)
}
}
if proxyReq.Header.Get("Accept") == "" {
proxyReq.Header.Set("Accept", "text/event-stream")
}
resp, respErr := httpClient.Do(proxyReq)
if respErr != nil {
c.JSON(http.StatusBadGateway, gin.H{"code": -1, "msg": "connect target failed: " + respErr.Error()})
return
}
defer resp.Body.Close()
forwardResponseHeaders(c.Writer.Header(), resp.Header)
c.Writer.WriteHeader(resp.StatusCode)
buf := make([]byte, 4096)
for {
n, readErr := resp.Body.Read(buf)
if n > 0 {
if _, writeErr := c.Writer.Write(buf[:n]); writeErr != nil {
return
}
c.Writer.Flush()
}
if readErr != nil {
return
}
}
}