From 84e3565de1f1a32723a50e5a4979aeaebf4f4d03 Mon Sep 17 00:00:00 2001 From: yuanyuanxiang <962914132@qq.com> Date: Mon, 24 Aug 2026 21:51:12 +0200 Subject: [PATCH 1/8] doc: Add remote control MCP design doc Add docs/Mcp_RemoteControl_Design.md, the reference design for an screenshot-driven AI remote-control feature: the existing get_screenshot for observation plus remote_open/remote_close/remote_mouse/remote_keyboard for input injection, reusing COMMAND_SCREEN_PREVIEW_REQ and COMMAND_SCREEN_CONTROL + MSG64. Coordinates are normalized (0..1) and mapped server-side to physical pixels; injection runs over the screen sub-connection opened by WebService::StartRemoteDesktop's hidden CScreenSpyDlg. Includes a chat-on-behalf experiment case with clipboard-encoding and window-capture notes. Also fold in the get_audit_log encoding correction for docs/Mcp_Terminal_Design.md, so the two doc changes ship as one commit. Co-Authored-By: deepseek-v4-pro --- docs/Mcp_RemoteControl_Design.md | 484 +++++++++++++++++++++++++++++++ docs/Mcp_Terminal_Design.md | 2 +- 2 files changed, 485 insertions(+), 1 deletion(-) create mode 100644 docs/Mcp_RemoteControl_Design.md diff --git a/docs/Mcp_RemoteControl_Design.md b/docs/Mcp_RemoteControl_Design.md new file mode 100644 index 0000000..39dd1c0 --- /dev/null +++ b/docs/Mcp_RemoteControl_Design.md @@ -0,0 +1,484 @@ +# MCP 远程控制（Remote Control）设计文档 + +> 版本：v1（设计基线）　状态：待开发　关联：`docs/Mcp_Design.md`、`docs/Mcp_Terminal_Design.md` +> 定位：本文件是「AI 远程控制一台 Windows 客户端（像人一样点鼠标、敲键盘、看画面完成任务）」功能的**唯一参照设计**，后续开发一律以本文为准，改需求先改本文。 + +--- + +## 1. 背景与目标 + +### 1.1 问题 + +现有 MCP 工具只能做「文本/命令行」层面的操作（`exec_command` 一次性命令、`terminal_*` 持久终端、`list_files`/`get_screenshot` 等只读）。很多真实任务只能在 **GUI 上完成**：点按钮、填表单、拖拽窗口、操作只有图形界面的软件。 + +AI 无法「看懂」视频流。把 25fps 的远程画面直接丢给视觉模型是行不通的——每秒 25 张、每张数十万像素，海量且绝大部分帧对决策无用。 + +### 1.2 目标 + +提供一组 MCP 工具，让 AI 以**「截图 → 观察 → 决策 → 注入单个输入动作 → 等待 → 重复」**的闭环驱动远程桌面： + +- **Observe（看）**：按需取一帧解码后的位图（JPEG），按视觉模型能力缩放到合适尺寸。 +- **Act（动）**：注入鼠标/键盘事件，复用现有远程桌面的输入注入链路。 +- **Decide（想）**：由 AI/宿主（Claude Code / 任何 MCP 客户端）在工具返回结果之上编排循环——**YAMA 只提供原语，不提供黑盒 `perform_task`**（与终端设计同一哲学）。 + +### 1.3 关键设计决策（已确认） + +| # | 决策 | 理由 | +|---|---|---| +| D1 | **截图驱动，非视频流** | 每次决策一张图，3–8s/步；视觉模型天然面向静态图 | +| D2 | **原语 + AI 编排，非黑盒任务** | 可控、可审、可中断；AI 在循环外可插判断/确认 | +| D3 | **归一化坐标（0..1）** | 彻底规避「截图缩放 vs 注入物理像素」的比例错位（最佳实践第一大坑） | +| D4 | **复用现有注入链路** | `COMMAND_SCREEN_CONTROL` + `MSG64` + 客户端 `SendInput` 已成熟 | +| D5 | **Observe 复用 `get_screenshot`** | 主连接按需抓 JPEG，无需常驻子连接，最省 | +| D6 | **独立开关 + 全程审计** | 与终端同级：`McpRemoteControl=1 && McpReadonly=0`，默认关 | + +--- + +## 2. 行业最佳实践借鉴 + +本节是「科学设计」的依据，逐条落到下文对应章节。主要参考 Anthropic 官方 Computer Use / Browser Use 最佳实践与参考实现（见文末「参考」）。 + +| 最佳实践 | 本设计落地 | +|---|---| +| 视觉模型是「规划者+眼睛」，宿主是「手」；工具调用=动作，返回=截图 | §3 循环归属：YAMA=手，AI=规划者 | +| **客户端先自行缩图**，绝不让模型 API 静默缩放（否则坐标系统性偏移） | §5：客户端按 `max_width` 缩图后 JPEG 回传，模型见到的就是声明尺寸 | +| 声明 `display_width/height` 必须 == 实际发送的图尺寸 | §7：归一化坐标，服务器侧映射物理像素，不依赖声明尺寸 | +| 动作原语集合（move/click/drag/scroll/type/key/…） | §4：`remote_mouse`/`remote_keyboard` 的动作枚举对齐 | +| 文本指令放在截图**之前**（内容顺序） | §8/宿主侧建议：由 MCP 客户端保证，本文仅记录 | +| 缩图上限：长边 ~1568px、总像素 ~1.15MP（Claude 4.6 系）；推荐 1280×720 | §5.3：`max_width` 默认建议 1280，可配 | +| 输入注入 vs 高危动作：不可逆操作应人工确认（human-in-the-loop） | §9.4：审计 + 宿主确认，YAMA 不做每动作确认（防延迟） | +| 提示注入是首要风险（模型处理不可信 UI 文本） | §9.5：全审计 + 只读关才可用 + 建议宿主隔离 | +| 长会话上下文管理（滚动缓冲/压缩） | §3：宿主职责，本文提示 | + +--- + +## 3. 总体架构 + +``` +┌──────────────────── 宿主（Claude Code / MCP 客户端）────────────────────┐ +│ 系统提示(含远程控制约定) + 任务文本 │ +│ ↓ │ +│ ┌──────────────────────── 智能体循环 ─────────────────────────┐ │ +│ │ get_screenshot(max_width) ──→ 视觉模型看图 ──→ 决策 ──→ │ │ +│ │ remote_mouse/remote_keyboard(归一化坐标/文本) ──→ 等待0.5–2s │ │ +│ │ 循环，直到任务完成或模型判定需用户介入 │ │ +│ └────────────────────────────────────────────────────────────┘ │ +└───────────────────────────────┬───────────────────────────────────────┘ + │ JSON-RPC over HTTP POST /mcp (Bearer) +┌───────────────────────────────▼───────────────────────────────────────┐ +│ YAMA 服务器（McpServer.cpp） │ +│ · get_screenshot：COMMAND_SCREEN_PREVIEW_REQ → 客户端缩图 JPEG │ +│ · remote_open：COMMAND_SCREEN_SPY → 建立屏幕子连接（控制会话） │ +│ · remote_mouse/remote_keyboard：归一化→物理像素 → MSG64 → │ +│ COMMAND_SCREEN_CONTROL（经屏幕子连接） │ +└───────────────────────────────┬───────────────────────────────────────┘ + │ 主连接（控制）+ 屏幕子连接（注入） +┌───────────────────────────────▼───────────────────────────────────────┐ +│ 客户端（ClientDll / ScreenManager / ScreenPreview） │ +│ · CaptureAndEncodePreview：抓主屏 + StretchBlt 缩图 + JPEG │ +│ · ScreenManager.ProcessCommand：解码 MSG64 → SendInput │ +└───────────────────────────────────────────────────────────────────────┘ +``` + +**核心事实（已核实，见 §13 源码索引）**： + +- **Observe** 走**主连接**、无子连接、无需控制模式：`COMMAND_SCREEN_PREVIEW_REQ`(247) → 客户端抓主屏缩图 JPEG → `TOKEN_SCREEN_PREVIEW_RSP`(248)。现有 `get_screenshot` 已完整实现。 +- **Act（注入）必须走屏幕子连接**：`COMMAND_SCREEN_CONTROL`(20) 只能经 `COMMAND_SCREEN_SPY`(16) 建立的子连接发送（`hub.go` 注释：input events MUST go through the sub-connection）。故需要一个 `remote_open` 建立/持有该子连接。 +- 服务器端同时维护一张**原始 BGRA 32bpp DIB**（`m_BitmapData_Full`），但 MCP 的 `get_screenshot` **不使用**它——它拿的是客户端已编码的 JPEG。两者坐标空间不同，见 §7。 + +--- + +## 4. 工具集设计 + +### 4.1 工具清单 + +| 工具 | 作用 | 入参（required 标注） | 出参 | +|---|---|---|---| +| `get_screenshot`（**已有，复用**） | Observe | `id`；可选 `max_width`(64..1920) | JPEG 图片（见 §5） | +| `remote_open` | 建立控制会话（屏幕子连接） | `id`；可选 `timeout_ms` | `session_id`(32hex)、`screen_w`/`screen_h`(物理虚拟桌面像素) | +| `remote_close` | 关闭会话 | `id`、`session_id` | `closed`(bool) | +| `remote_mouse` | 注入鼠标 | `id`、`session_id`、`action`、`x`、`y`（+ 按 action 需 `x2`/`y2`/`button`/`delta`/`clicks`） | `{}`（成功即空对象） | +| `remote_keyboard` | 注入键盘 | `id`、`session_id`、`action`（+ 按 action 需 `text`/`key`/`modifiers`） | `{}` | + +> 命名对齐现有 `terminal_*`（open/exec/close），Observe 沿用 `get_screenshot` 避免重复造轮子。鼠标/键盘各一个工具、用 `action` 枚举区分具体动作——既对齐 Anthropic 的 `computer` 工具动作集，又贴合现有 `WebService::HandleMouse/HandleKey` 的「鼠标/键盘分家」实现。 + +### 4.2 `remote_mouse` 动作枚举 + +| action | 参数 | 语义 | +|---|---|---| +| `move` | `x`,`y` | 移动光标（不按键） | +| `down` | `x`,`y`,`button`(left/middle/right) | 按下 | +| `up` | `x`,`y`,`button` | 抬起 | +| `click` | `x`,`y`,`button`(默认 left)、`clicks`(1/2/3，默认 1) | 单击/双击/三击 | +| `right_click` | `x`,`y` | 右键（`click` 的便捷别名） | +| `middle_click` | `x`,`y` | 中键 | +| `drag` | `x`,`y`,`x2`,`y2`,`button`(默认 left) | 从 (x,y) 拖到 (x2,y2) | +| `scroll` | `x`,`y`,`delta`(+/- 滚动量)、`button`(可选，垂直默认) | 滚轮 | + +### 4.3 `remote_keyboard` 动作枚举 + +| action | 参数 | 语义 | +|---|---|---| +| `type` | `text` | 输入一串文本（字符级映射，非粘贴） | +| `key_down` | `key`、`modifiers`(可选) | 按下组合键（如 `Ctrl`+`C`） | +| `key_up` | `key`、`modifiers`(可选) | 抬起 | +| `key_press` | `key`、`modifiers`(可选) | 按下并抬起（最常用） | + +`key` 取值对齐 Windows 虚拟键名（`VK_*` 去掉前缀，如 `ENTER`/`TAB`/`F5`/`LEFT`）与常见修饰键 `CTRL`/`ALT`/`SHIFT`/`WIN`。`type` 文本中的字符经客户端 `VkKeyScan`/Unicode 注入映射为键盘事件（非剪贴板粘贴，保证焦点内输入）。 + +### 4.4 可选/扩展工具（`remote_clipboard`、`remote_focus_window` 为 §17 代聊实验所需，建议随阶段一落地） + +- `remote_clipboard`：写远程剪贴板（UTF-8→Unicode），非 ASCII 文本输入的关键路径；复用 `COMMAND_C2C_TEXT`(90) 的 UTF-8 逻辑或新增 UTF-8 设剪贴板命令（§6.4）。 +- `remote_focus_window`：`COMMAND_SCREEN_WINDOW`(153) / 客户端 `SetForegroundWindow`，把焦点给到指定窗口（配合 `list_windows` 的 `hwnd`）；代聊第一步。 +- `remote_cursor_position`：读当前光标（`TOKEN_NEXTSCREEN` 帧头已带 `POINT cursor`）。 +- `remote_screenshot`（会话内/窗口抓屏）：从子连接 DIB 服务端缩图，或按窗口抓取（`COMMAND_SCREEN_WINDOW`），供高分辨率读聊天文字与多显示器场景（§12 阶段二、§17）。 + +--- + +## 5. 屏幕捕获与编码（Observe） + +### 5.1 现有链路（复用） + +`get_screenshot`（`McpServer.cpp` `BuildGetScreenshot`，~1597）已实现： + +1. `ParseHostIdArg` → `FindMainContext` → 能力门 `ctx->SupportsScreenPreview()`（`CLIENT_CAP_SCREEN_PREVIEW`，不支持 → `-32005`）。 +2. `BeginPending(devId,"get_screenshot")` 设备级互斥（忙 → `-32003`）。 +3. `reqId = NextPreviewReqId()`；`SetPendingReqId(devId, reqId)`（stale-drop 机制）。 +4. `parent->ChooseScreenPreviewParams(...)` 按 RTT/FRP 自适应缩图档位 + `max_width` 覆盖（clamp 64..1920）。 +5. `SendScreenPreviewRequest` → `COMMAND_SCREEN_PREVIEW_REQ`。 +6. `WaitPending` 收 `[ScreenPreviewRspHeader][JPEG]`，校验 `status==OK && format==JPEG`。 +7. base64 JPEG 返回：`structuredContent.image = {mimeType:"image/jpeg", width, height, bytes}`，`content=[{type:"image", data:, mimeType:"image/jpeg"}]`。 + +客户端 `CaptureAndEncodePreview`（`client/ScreenPreview.cpp:107`）抓**主屏** + `StretchBlt`(HALFTONE) 缩图 + GDI+ `Bitmap::Save` JPEG——**位图永不离开客户端，只回传 JPEG**。 + +### 5.2 缩放策略（对齐最佳实践 D5） + +- 缩图在**客户端**完成（`max_width` 长边约束），服务器/MCP 宿主**不再二次缩放**，模型看到的就是声明尺寸——避免「模型 API 静默缩放导致坐标漂移」这个头号坑。 +- `max_width` 由宿主按视觉模型能力显式传入；YAMA 提供**默认建议 1280**（长边），上限维持 1920。 + +### 5.3 推荐参数 + +| 项 | 建议值 | 说明 | +|---|---|---| +| `max_width`（长边） | 1280（默认）/ 1568（上限） | 1280×720 ≈ 80% 像素预算、训练常见分辨率 | +| 编码 | JPEG，quality ~70–85 | 客户端 `jpegQuality` 已有，按 `ChooseScreenPreviewParams` 自适应 | +| 返回 | base64 `image/jpeg` | 已实现 | + +--- + +## 6. 输入注入（Act） + +### 6.1 协议（复用，已核实） + +- **包格式**：`[cmd:1][MSG64:48]`，可批量重复（`commands.h` `COMMAND_SCREEN_CONTROL=20`，`MSG64` 1581–1623）。 +- **MSG64 布局**：`{uint64 hwnd, message, wParam, lParam, time; POINT pt;}` = 48 字节（`_WIN64` 下 `MYMSG` 即 `MSG`；客户端亦接受 28 字节 `MSG32`，由 `ulLength % 28/48` 判定）。 +- **客户端解码**：`ScreenManager.cpp:1175` → `ProcessCommand(1727)` → `SendInput`： + - 鼠标：坐标取 `LOWORD/HIWORD(lParam)`，`WM_MOUSEMOVE`→`MOUSEEVENTF_MOVE`，`WM_LBUTTONDOWN/UP`→`LEFTDOWN/LEFTUP`，`WM_LBUTTONDBLCLK`→额外 `LEFTDOWN`，`WM_MBUTTONDOWN/UP`，`WM_MOUSEWHEEL`→`MOUSEEVENTF_WHEEL`(`mouseData=GET_WHEEL_DELTA_WPARAM`)。 + - 键盘：`wVk=(WORD)wParam`，`wScan=(lParam>>16)&0xFF`，`KEYEVENTF_EXTENDEDKEY` 由 `(lParam>>24)&1`。 + +### 6.2 服务端可复用实现 + +`WebService::HandleMouse`（`WebService.cpp:773-865`）与 `HandleKey`（`867-952`）已从 JSON（`type/x/y/button/delta`；`keyCode/down/altKey`）构造 `MSG64` 并 `[COMMAND_SCREEN_CONTROL][MSG64]` 发送。其中 `HandleKey` 已正确处理 `lParam`：repeat=1、`MapVirtualKey` 扫瞄码、扩展键表、Alt 上下文位 29、keyup 位 30–31。 + +**结论**：`remote_mouse`/`remote_keyboard` 是这两个函数的 MCP 薄封装——先归一化→物理像素，再套用其 `MSG64` 构造逻辑。**不要新写注入协议**。 + +### 6.3 子连接前置条件（关键约束） + +注入**必须**经屏幕子连接： + +1. 主连接发 `COMMAND_SCREEN_SPY`(16)（载荷 `{cmd, USING_DXGI|2, ALGORITHM_*, MultiScreen}`，见 `2015RemoteDlg.cpp:8443-8470`）→ 客户端开子连接（`ClientDll.cpp:89-94`）。 +2. 客户端回 `TOKEN_BITMAPINFO`（含物理 `BITMAPINFOHEADER`：`biWidth/biHeight`）——**这是归一化坐标映射所需的物理分辨率来源**。 +3. 服务器发 `COMMAND_NEXT`(30) 开始流/允许控制。 +4. `GetScreenContext(device_id)`（`WebService.cpp:2084`）从 `m_ScreenContexts` 取该子连接上下文。 + +> 实现提示：上述 1–4 的建立/持有逻辑**耦合在 `CScreenSpyDlg`（隐藏对话框）里**——子连接、帧 DIB、`TOKEN_BITMAPINFO` 处理都由它完成。MCP 的 `remote_open` 应复用 `WebService::StartRemoteDesktop`(1725) 的「隐藏对话框」路径，而非另造无对话框持有者（§13.2）。 + +`remote_open` 即上述 1–4 的封装；`screen_w/screen_h` 取自 `TOKEN_BITMAPINFO`。 + +### 6.4 文本输入：ASCII 直注 vs 非 ASCII 走剪贴板 + +`remote_keyboard` 的 `type` 走物理键事件（`wVk`/`wScan` → `SendInput`），只能可靠注入 ASCII/ANSI 字符——**中文等非 ASCII 文本无法靠模拟键盘键入**（需 IME 组合，键事件层面做不到）。 + +非 ASCII 文本的可靠路径是**剪贴板 + 粘贴**： + +- 服务器设远程剪贴板命令 `COMMAND_SCREEN_SET_CLIPBOARD`(25)，包 `[cmd:1][text:N]`，经屏幕子连接发送；客户端 `UpdateClientClipboard`（`ScreenManager.cpp:1223/1599`）落到剪贴板。 +- 随后 `remote_keyboard` 注入 `Ctrl+V` 粘贴、`Enter` 发送。 +- **编码注意（已核实）**：`UpdateClientClipboard` 用 `SetClipboardData(CF_TEXT,...)`——ANSI/GBK 格式（`CF_TEXT`，非 `CF_UNICODETEXT`）。MVP 下中文可经 `ToAnsi(utf8, 936)` 转 GBK 后直发（与终端命令同模式）；但 emoji/非 GBK 字符会丢、部分新式聊天软件粘贴偏好 Unicode。更稳做法是复用 `COMMAND_C2C_TEXT`(90) 的 UTF-8→Unicode→`CF_UNICODETEXT` 逻辑（`KernelManager.cpp:1573-1587`，且已内置「设完剪贴板后自动模拟 Ctrl+V」），或新增一个 UTF-8 设剪贴板变体。见 §17 实验案例。 + +--- + +## 7. 坐标约定与映射 + +### 7.1 归一化坐标（D3） + +- AI 在**所有鼠标动作**里输出 **0..1 浮点**：`x = 目标横向比例，y = 目标纵向比例`，相对**它看到的那张截图**。 +- 服务器侧映射：`phys_x = round(norm_x * screen_w)`、`phys_y = round(norm_y * screen_h)`，其中 `screen_w/h` 是 `remote_open` 时从 `TOKEN_BITMAPINFO` 得到的**物理捕获分辨率**。 +- 好处：无论 `max_width` 怎么缩、显示器多大，AI 永远只说「在图的 40%/60% 处点一下」，比例错位风险为零。 + +### 7.2 坐标空间一致性（最重要正确性点） + +Observe（`get_screenshot` 抓**主屏**）与 Act（`SendInput` 绝对坐标在**虚拟桌面**空间）默认不同源。**阶段一只支持单显示器**：主屏 == 虚拟桌面，两者重合，映射严格正确。 + +多显示器（`MultiScreen`）在**阶段二**处理：届时 Observe 必须改抓**整个虚拟桌面**（而非主屏），`screen_w/h` 用虚拟桌面尺寸，归一化坐标才继续成立。**在阶段二落地前，`remote_open` 对多显示器配置应返回显式错误或降级为主屏**（见 §12）。 + +### 7.3 边界与钳制 + +- 归一化值越界（`<0` 或 `>1`）→ 钳制到 `[0,1]` 或返回 `-32602`（建议钳制，模型偶发 `1.0001`）。 +- `phys_x/y` 不得越过 `screen_w/h`，注入前 `clamp(0, screen_w-1)`。 + +--- + +## 8. 控制会话状态机 + +### 8.1 生命周期 + +``` +remote_open ──► 建立子连接 ──► [remote_mouse/remote_keyboard × N + get_screenshot × M] + │ │ + └──（busy 互斥 / idle 回收 / 断线清理）──────────┴──► remote_close（幂等） +``` + +### 8.2 状态与不变量（镜像 `docs/Mcp_Terminal_Design.md` §并发） + +- **单设备单控制会话**：`remote_open` 对已存在会话 → `-32003`（忙）。 +- **互斥**：同一设备上，MCP 远程控制会话与**人类远程桌面观看**（`m_ScreenContexts` 已被占用）互斥——`remote_open` 发现已有屏幕子连接 → `-32003`，避免 AI 向人类正在观看/控制的画面注入。`remote_mouse/keyboard` 要求 `session_id` 匹配且会话未关闭。 +- **busy 标志**：一条注入在飞时 `busy=true`，保证等待线程是唯一擦除者（沿用终端的 `m_TermMutex`+`busy` 模式；远程控制建议独立 `m_ScreenCtrlMutex`/`m_ScreenCtrlCv`）。 +- **idle 回收**：会话闲置超时（默认 300s）自动关闭并 `CancelIO`（镜像 `SweepIdleTerminals`）。 +- **断线清理**：`OfflineProc`/`OnTerminalClosed` 同位置加 `McpServer().OnScreenControlClosed(subCtx)`，防悬空 `subCtx` 被连接池复用（复刻终端设计里发现的第 2 个修复点）。 +- **锁外 `CancelIO`**：所有取消动作在锁外执行（终端的 P2 教训）。 + +### 8.3 并发（P5 教训） + +`httplib` 多线程并发处理请求；`remote_mouse` 会高频调用。会话状态一律 `m_ScreenCtrlMutex` 串行化，**不要假定请求串行**。高频注入可加**节流**（沿用 `SendScaledMouseMessage` 的鼠标移动节流思路，`ScreenSpyDlg.cpp:2728-2743`）。 + +--- + +## 9. 安全模型 + +延续既有「可控、可关、可审、可隔离」原则（与终端同级）： + +### 9.1 开关与门控 + +- 新增 `McpRemoteControl`（默认 **0=关**）。 +- `remote_*` 生效条件：`McpRemoteControl=1 && McpReadonly=0`，否则 `-32006`。 +- `get_screenshot` 是只读工具，不受此门控，仍仅受现有能力门（供纯观察场景）。 + +### 9.2 全程审计 + +每条 `remote_open/close/mouse/keyboard` 调用 → `PostMessageA(WM_SHOWERRORMSG, ...)` → `m_MessageLog` → `get_audit_log` 可查（复用 `McpServer.cpp:2050` 的审计模式，标题 `MCP远程控制`，标记「不可关闭」）。审计内容含：设备 id、动作、归一化坐标与换算后的物理坐标、session_id、时间戳。 + +### 9.3 会话级鉴权 + +- `session_id` 由 `GenerateRandomToken()`（32hex），每次 `remote_open` 独立随机，防串用（镜像终端 P6）。 +- 注入工具校验 `session_id` 归属与设备匹配，不匹配 → `-32002`。 + +### 9.4 不可逆操作与人工确认 + +- YAMA **不做逐动作确认**（会引入每步 0.5s+ 延迟，破坏闭环节奏）。 +- 高危动作（提交表单、删除、付款、改系统设置）的确认由**宿主/AI 编排层**负责（Claude Code 的权限系统 / 系统提示约定「不可逆操作前暂停询问用户」）。本文记录此分工；YAMA 提供审计留痕兜底。 + +### 9.5 提示注入（Prompt Injection） + +视觉模型会读到客户端屏幕上的**不可信 UI 文本**，存在注入风险。缓解：远程控制本身即高危能力（默认关 + 只读关才开 + 全审计）；建议宿主将远程控制会话隔离在独立权限域，并对「屏幕文本里的指令」保持怀疑。YAMA 侧不做内容分类（无意义且加延迟），记录即可。 + +--- + +## 10. 错误码 + +沿用现有约定（`BuildError(id, code, msg)`）： + +| code | 含义 | 触发 | +|---|---|---| +| -32602 | 参数非法 | action 未知、必填参数缺失、`max_width` 越界、归一化值格式错 | +| -32000 | 通用失败 | 未分类错误 | +| -32001 | 超时 | `remote_open` 等子连接/`get_screenshot` 等帧超时 | +| -32002 | 设备/会话不存在 | `id` 无在线主机、`session_id` 不存在或不匹配 | +| -32003 | 设备/会话忙 | 已有控制会话、人类远程桌面占用、`get_screenshot` 在飞 | +| -32004 | 发送失败 | 注入命令发送失败 | +| -32005 | 能力不支持 | 客户端无 `CLIENT_CAP_SCREEN_PREVIEW`（Observe）/ 不支持屏幕控制（Act） | +| -32006 | 被开关禁用 | `McpRemoteControl=0` 或 `McpReadonly=1` | +| -32008 | 非法/被拒参数 | 多显示器未支持时的 `remote_open` 降级拒绝（阶段一） | + +--- + +## 11. 配置项 + +新增 1 项（`THIS_CFG` `settings` 节，键名沿用蛇形）： + +| 键 | 默认 | 说明 | +|---|---|---| +| `McpRemoteControl` | `0` | 远程控制开关；要求 `McpReadonly=0` 才生效 | + +其余沿用：`McpEnabled`/`McpPort`/`McpBind`/`McpToken`/`McpReadonly`/`McpTerminal`/`McpCmdWhitelist`。改动需重启生效（与终端一致）。 + +--- + +## 12. 已知限制与后续扩展 + +### 12.1 阶段一（本期） + +- **单显示器**：Observe 抓主屏，坐标空间与虚拟桌面重合；多显示器 `remote_open` 显式报错（`-32008`）。 +- 注入基于**绝对坐标** `SendInput`，要求客户端处于可注入的桌面会话（已登录、非锁屏/非 UAC 安全桌面）；锁屏/UAC 提示场景注入无效（客户端 `SendInput` 限制），设计文档记录为已知限制。 +- **代聊实验使能项**：窗口抓屏（`COMMAND_SCREEN_WINDOW`）、UTF-8 剪贴板、窗口聚焦随阶段一落地（§17）。 + +### 12.2 阶段二（后续） + +1. **多显示器虚拟桌面捕获**：`get_screenshot` 增加虚拟桌面抓屏，或新增 `remote_screenshot` 从子连接 DIB 服务端缩图（复用 `BmpToJpeg`/`CacheThumbnail` GDI+ 路径），使 Observe/Act 同源。 +2. **窗口定向**：`remote_focus_window`（复用 `COMMAND_SCREEN_WINDOW` + 客户端 `SetForegroundWindow`）。 +3. **光标读取 / 剪贴板**。 +4. **服务端缩图**：从 `m_BitmapData_Full`（BGRA DIB）直接按 API 上限缩图编码，进一步省客户端往返。 + +--- + +## 13. 改动清单（供实现参照） + +> 行号以当前树为准（实现前请 grep 复核）。新增/改动尽量「复用」而非「重写」。 + +### 13.1 `server/2015Remote/McpServer.h` + +- 新增 `struct ScreenCtrlSession`（仿 `TermSession`）：`sessionId`、`deviceId`、`busy`、`started`、`closed`、`lastActiveAt`、`screenW/screenH`（来自 `TOKEN_BITMAPINFO`）、`subCtx` 引用。 +- 新增方法：`SetRemoteControlEnabled/IsRemoteControlEnabled`、`BeginScreenCtrlOpen/…/CloseScreenCtrlSession/SweepIdleScreenCtrl`、`OnScreenControlClosed(context*)`、`ResolveScreenCtrlSessionHost`。 +- 新增成员：`bool m_remoteControlEnabled=false;` 及 `std::mutex m_ScreenCtrlMutex; std::condition_variable m_ScreenCtrlCv; std::map m_ScreenCtrlContextToDevice; std::map m_ScreenCtrlSessions;`（或复用 WebService 的 `m_ScreenContexts` 生命周期，见下）。 + +### 13.2 `server/2015Remote/McpServer.cpp` + +- 新增 4 个 schema 构建器 + 3 个 handler：`BuildRemoteOpen/Close/Mouse/Keyboard`（镜像 `BuildTerminalOpen` 等，`BuildToolsListResult` ~1278 后加条目，`BuildToolsCall` ~2124 加分派）。 +- `remote_open`：`SweepIdleScreenCtrl` → `ResolveScreenCtrlSessionHost` → 已有会话 `-32003` → `GenerateRandomToken` → **复用 Web 远程桌面的会话建立路径 `WebService::StartRemoteDesktop`(1725)**（发 `COMMAND_SCREEN_SPY` → 以 `SW_HIDE` 打开隐藏 `CScreenSpyDlg` 持有子连接 → 客户端子连接到达 → `RegisterScreenContext` 记入 `m_ScreenContexts`）→ 等 `TOKEN_BITMAPINFO` 取 `screenW/H` → `COMMAND_NEXT` → 审计 → 返回 `{session_id, screen_w, screen_h}`。**注意：屏幕子连接/帧 DIB/`TOKEN_BITMAPINFO` 处理都与 `CScreenSpyDlg` 耦合，不要另造无对话框持有者——沿 Web 的隐藏对话框路径是低风险正解。** +- `remote_mouse/keyboard`：`ResolveScreenCtrlSessionHost` → 校验 `session_id`/`busy` → 归一化→物理像素（`round(norm * screenW/H)` + clamp）→ **复用 `WebService::HandleMouse/HandleKey` 的 `MSG64` 构造**（或抽取成共享 helper `BuildMouseMsg64/BuildKeyMsg64`，避免 MCP 与 Web 两处重复）→ `subCtx->Send2Client([COMMAND_SCREEN_CONTROL][MSG64])` → 审计。 +- `remote_close`：幂等 `CloseScreenCtrlSession`（不存在 → `{closed:true}`；sid 不匹配 → `-32002`）→ 锁外 `CancelIO` → 审计。 +- `OnScreenControlClosed`：镜像终端修复点——空闲会话直接擦路由+会话，busy 会话唤醒等待线程由其清理。 +- `SweepIdleScreenCtrl`：镜像 `SweepIdleTerminals`（`difftime` 防时钟回拨）。 + +### 13.3 `server/2015Remote/McpSettingsDlg.h/.cpp` + +- 新增复选框 `IDC_MCP_REMOTECONTROL = 1008`，文案 `_TR("启用远程控制（AI 操控桌面）")`；回填/落盘 `McpRemoteControl`；对话框高度 +30。 + +### 13.4 `server/2015Remote/2015RemoteDlg.cpp` + +- 启动读配置处加 `McpServer().SetRemoteControlEnabled(THIS_CFG.GetInt("settings","McpRemoteControl",0)!=0);`。 +- `OfflineProc` 加 `if (McpServer().IsRunning() && McpServer().IsScreenCtrlContext(ContextObject)) McpServer().OnScreenControlClosed(ContextObject);`（复刻终端修复点）。 + +### 13.5 语言文件 + +- 沿用终端约定：MCP 相关文案本就在 `lang/*.ini` 无条目（`_TR` 对未命中键原样返回中文）。若日后翻译，用 **GBK 工具**改 `lang/{en_US,zh_TW}.ini`（ANSI/GBK，**不能用 Write/Edit 直接写**）。 + +--- + +## 14. 关键点（实现时务必遵守） + +- **P1（坐标一致性，最重要）**：归一化坐标映射所用 `screenW/H` 必须与**当前截图来源的分辨率**一致。阶段一单屏下 `TOKEN_BITMAPINFO` 的主屏尺寸 == 截图主屏物理尺寸，才成立；任何「用虚拟桌面尺寸映射主屏截图」都会造成系统性偏移（最佳实践第一大坑）。 +- **P2（注入走子连接）**：`COMMAND_SCREEN_CONTROL` **必须**经 `GetScreenContext` 取得的子连接发送，主连接无效。 +- **P3（锁外 CancelIO）**：所有 `CancelIO` 在 `m_ScreenCtrlMutex` 之外（终端 P2 教训）。 +- **P4（复用勿重写）**：`MSG64` 构造逻辑复用 `WebService::HandleMouse/HandleKey`；理想做法抽共享 helper，否则未来两处漂移。 +- **P5（并发串行化）**：`remote_mouse` 高频并发，状态全部走锁；勿假定请求串行（终端 P5 教训）。 +- **P6（session_id 独立随机 + 互斥）**：会话 token 用 `GenerateRandomToken`；单设备单会话；与人类远程桌面观看互斥（`-32003`）。 +- **P7（非黑盒）**：不实现 `perform_task`；循环归宿主/AI，YAMA 只出原语（D2）。 + +--- + +## 15. 验证（端到端，127.0.0.1:6544，Bearer `a2176213eddcf0acfdb285940ec4eb2a`） + +前置：设置开 `McpEnabled=1`、`McpReadonly=0`、`McpRemoteControl=1`；目标 Windows 客户端在线、已登录桌面。`id` 取 `list_online_hosts` 的十进制主机 id。 + +```bash +B='Authorization: Bearer a2176213eddcf0acfdb285940ec4eb2a' +U=http://127.0.0.1:6544/mcp + +# 1) tools/list 应出现 remote_open/close/mouse/keyboard 四个工具 +curl -s $U -H "$B" -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' + +# 2) 打开会话 → session_id + screen_w/h +curl -s $U -H "$B" -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"remote_open","arguments":{"id":""}}}' + +# 3) Observe：截图（复用） +curl -s $U -H "$B" -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"get_screenshot","arguments":{"id":"","max_width":1280}}}' + +# 4) Act：移动 + 单击（归一化坐标，0.5,0.5 = 屏幕中心） +curl -s $U -H "$B" -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"remote_mouse","arguments":{"id":"","session_id":"","action":"move","x":0.5,"y":0.5}}}' +curl -s $U -H "$B" -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"remote_mouse","arguments":{"id":"","session_id":"","action":"click","x":0.5,"y":0.5,"button":"left"}}}' + +# 5) 键盘：按下并抬起 Ctrl+Esc 或输入文本 +curl -s $U -H "$B" -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"remote_keyboard","arguments":{"id":"","session_id":"","action":"key_press","key":"WIN"}}}' + +# 6) 负例：未开会话/错误 sid → -32002；已开会话再 open → -32003；只读/开关关 → -32006 +# 7) 关闭 + 幂等关闭 +curl -s $U -H "$B" -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":9,"method":"tools/call","params":{"name":"remote_close","arguments":{"id":"","session_id":""}}}' + +# 8) 审计链 +curl -s $U -H "$B" -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":12,"method":"tools/call","params":{"name":"get_audit_log","arguments":{}}}' +``` + +人工核对：注入后目标机光标/焦点确实变化；审计列表出现标题「MCP远程控制」的 open/close/mouse/keyboard 条目；闲置 >300s 后旧 `session_id` 调用 → `-32002`。编译由你以 VS2019（MSBuild v143）完成；我侧仅代码审查（工具集 v142，无法在此构建）。 + +--- + +## 16. 涉及文件 + +- `server/2015Remote/McpServer.h` / `.cpp`（会话状态机 + schema + handler + 分派） +- `server/2015Remote/McpSettingsDlg.h` / `.cpp`（新开关） +- `server/2015Remote/2015RemoteDlg.cpp`（启动读配置 + `OfflineProc` 清理钩子） +- `common/commands.h`（**只读参照**：`COMMAND_SCREEN_SPY`/`COMMAND_SCREEN_CONTROL`/`MSG64`/`COMMAND_SCREEN_PREVIEW_REQ` 等，无需改） +- `server/2015Remote/WebService.cpp`（**复用参照**：`HandleMouse`/`HandleKey`/`GetScreenContext`） +- `lang/{en_US,zh_TW}.ini`（可选，GBK 工具改） + +--- + +## 17. 实验案例：AI 代聊（chat-on-behalf） + +> 目标：以「AI 通过远程桌面，在聊天软件里代表你与朋友聊天」为第一个实验用例，验证整条「看 → 想 → 动 → 确认」链路。低风险、目标清晰、信号明确（有新消息 / 已发出）。 + +### 17.1 闭环 + +1. `list_windows` 找到聊天窗口（`hwnd`/`title`）。 +2. `remote_focus_window`（或 `remote_mouse` 点击输入框）聚焦聊天窗。 +3. 抓**聊天窗口**高分辨率截图 → 视觉模型读出朋友最新消息。 +4. 模型生成回复。 +5. `remote_clipboard` 把回复写入远程剪贴板（UTF-8）。 +6. `remote_keyboard` 注入 `Ctrl+V`（粘贴）+ `Enter`（发送）。 +7. 再截图确认已发出，等待下一条。 + +### 17.2 两个已验证的关键依赖 + +- **中文输入（走剪贴板，已解决）**：见 §6.4。`COMMAND_SCREEN_SET_CLIPBOARD`(25) 为 ANSI/GBK（`CF_TEXT`），中文可经 `ToAnsi(utf8, 936)` 直发跑通 MVP；要支持 emoji/完整 Unicode，复用 `COMMAND_C2C_TEXT`(90) 的 UTF-8→`CF_UNICODETEXT` 逻辑或新增 UTF-8 变体。客户端 `KernelManager.cpp:1590` 已内置「设剪贴板后自动模拟 Ctrl+V」。 +- **读消息（需窗口抓屏）**：`get_screenshot` 抓整屏，缩到 1280 后聊天文字过小、OCR 易错。用 `COMMAND_SCREEN_WINDOW`(153) 按窗口抓取聊天窗（更高有效分辨率），或对该区域用更高 `max_width`。 + +### 17.3 使能清单（相对现有代码的增量） + +| 能力 | 状态 | 复用点 | +|---|---|---| +| `get_screenshot` / `list_windows` | ✅ 已有 | — | +| 剪贴板写入 | ⚠️ 客户端已有（GBK 直发可跑通），缺 MCP 封装 | `COMMAND_SCREEN_SET_CLIPBOARD`(25) / `COMMAND_C2C_TEXT`(90) | +| `remote_open` / `remote_close`（屏幕子连接） | ❌ 待建 | `COMMAND_SCREEN_SPY`(16) 建立逻辑 `2015RemoteDlg.cpp:8443` | +| `remote_mouse` / `remote_keyboard` | ❌ 待建 | `WebService::HandleMouse/HandleKey` | +| `remote_focus_window` | ❌ 待建 | `COMMAND_SCREEN_WINDOW`(153) | +| `remote_clipboard` | ❌ 待建 | 见 §6.4 | + +### 17.4 最小 MVP 实施顺序（按依赖） + +1. `remote_open` / `remote_close`（屏幕子连接 + `TOKEN_BITMAPINFO` 取分辨率）。 +2. `remote_keyboard`（先只做 `key_press`/`type` + 修饰键，够发 `Ctrl+V`/`Enter`）。 +3. `remote_mouse`（`click`/`move`，用于点输入框；聚焦可用点击替代）。 +4. `remote_clipboard`（先 GBK 直发跑通，再上 UTF-8 变体）。 +5. `remote_focus_window`（或用步骤 3 的点击替代，二选一即可跑通）。 + +> 1→2→3 已足以发出一条消息（点输入框 + 粘贴 + 回车）；4 是中文关键路径；5 是体验优化，可后置。 + +### 17.5 实验注意事项 + +- 目标机必须**已登录且未锁屏**（锁屏/UAC 安全桌面下抓屏与注入均失效，§12.1）。 +- 「代聊」会让朋友误以为是真人在聊——实验前建议告知对方；避免让 AI 在承诺/隐私等话术上自由发挥（§9.4 的人工确认原则同样适用）。 +- 新消息检测先用「轮询截图」即可（MVP 接受），后续用未读角标/窗口通知优化（§17.6）。 + +### 17.6 后续优化（非 MVP） + +- 新消息触发：从「定时截图」改为「未读角标/窗口通知驱动」，大幅省 token。 +- 窗口定向抓屏（`COMMAND_SCREEN_WINDOW`）并入 `remote_screenshot`。 +- 会话历史：AI 保留最近几轮聊天上下文，避免回复脱节（宿主侧滚动缓冲，§3）。 + +--- + +## 参考（行业最佳实践） + +- Anthropic《Best practices for computer and browser use with Claude》— https://claude.com/fr/blog/best-practices-for-computer-and-browser-use-with-claude +- Anthropic 参考实现 `computer.py`（动作枚举、缩放、坐标换算）— https://github.com/anthropics/claude-quickstarts/blob/main/computer-use-demo/computer_use_demo/tools/computer.py +- 《How Claude Computer Use Works: Architecture Internals》— https://callsphere.ai/blog/how-claude-computer-use-works-architecture-internals diff --git a/docs/Mcp_Terminal_Design.md b/docs/Mcp_Terminal_Design.md index e73c744..c6be08e 100644 --- a/docs/Mcp_Terminal_Design.md +++ b/docs/Mcp_Terminal_Design.md @@ -451,7 +451,7 @@ McpServer().SetTerminalEnabled(THIS_CFG.GetInt("settings", "McpTerminal", 0) != ### 10.6 中文/乱码 - 输出乱码：检查 `isPty` 与 `cp` 是否匹配（PTY=UTF-8，老管道=GBK）。 -- **`get_audit_log` 中文乱码**：已知问题（GBK 写入 UTF-8 JSON），影响**所有**审计条目，非终端特有，暂未修复。 +- **`get_audit_log` 中文正确**：`BuildGetAuditLog` 用 `ToUtf8(type/time/msg, 936)` 把 GBK 转 UTF-8，jsoncpp 输出 `\uXXXX` 转义，客户端解析后中文正常（已实测 `MCP持久终端`/`MCP命令执行`/`操作成功`/`主机上线` 等均无乱码）。 ### 10.7 会话泄漏 / 子链接不关 -- 2.43.0 From 6045baadb86daf3be18d006c5f671e7efce104d2 Mon Sep 17 00:00:00 2001 From: yuanyuanxiang <962914132@qq.com> Date: Tue, 25 Aug 2026 12:55:52 +0200 Subject: [PATCH 2/8] Feature: Add MCP remote_open/remote_close remote control sessions Add M1 of MCP remote control (docs/Mcp_RemoteControl_Design.md): the remote_open / remote_close tools plus the ScreenCtrlSession state machine. remote_open establishes a hidden screen sub-connection by reusing WebService::StartRemoteDesktop (COMMAND_SCREEN_SPY -> CScreenSpyDlg -> RegisterScreenContext), polls for the sub-connection plus its physical resolution (TOKEN_BITMAPINFO -> NotifyResolutionChange -> GetScreenSize), then records the session (single device, single session, reverse-mapped subCtx for OfflineProc cleanup) and returns {session_id, screen_w, screen_h}. remote_close validates session_id and tears down the sub-connection idempotently. A McpRemoteControl settings checkbox (default off, requires McpReadonly=0) gates the tools; every open/close is audited via WM_SHOWERRORMSG. Gating: multi-monitor hosts are rejected with -32008 (phase 1 supports only single monitor, where Observe=main screen and Act=virtual desktop coincide); the monitor count comes from the client heartbeat RES_RESOLUTION ("N:W*H"). Known limitations (deferred to the injection milestones): mutual exclusion with human remote-desktop viewing is one-directional in M1 (a human who joins during an MCP session can tear it down on disconnect), and subCtx is not yet dereferenced so no liveness re-check is needed until remote_mouse/remote_keyboard. Co-Authored-By: deepseek-v4-pro --- server/2015Remote/2015RemoteDlg.cpp | 7 + server/2015Remote/McpServer.cpp | 388 +++++++++++++++++++++++++++ server/2015Remote/McpServer.h | 44 ++- server/2015Remote/McpSettingsDlg.cpp | 17 +- server/2015Remote/McpSettingsDlg.h | 2 + server/2015Remote/WebService.cpp | 11 + server/2015Remote/WebService.h | 4 + 7 files changed, 469 insertions(+), 4 deletions(-) diff --git a/server/2015Remote/2015RemoteDlg.cpp b/server/2015Remote/2015RemoteDlg.cpp index 2ce018c..c85cc56 100644 --- a/server/2015Remote/2015RemoteDlg.cpp +++ b/server/2015Remote/2015RemoteDlg.cpp @@ -2195,6 +2195,8 @@ BOOL CMy2015RemoteDlg::OnInitDialog() McpServer().SetCmdWhitelist(THIS_CFG.GetStr("settings", "McpCmdWhitelist", "")); // 持久终端开关：默认关；要求只读关（McpReadonly=0）才生效（工具列表/分派双重门控）。 McpServer().SetTerminalEnabled(THIS_CFG.GetInt("settings", "McpTerminal", 0) != 0); + // 远程控制开关：默认关；要求只读关（McpReadonly=0）才生效（工具列表/分派双重门控）。 + McpServer().SetRemoteControlEnabled(THIS_CFG.GetInt("settings", "McpRemoteControl", 0) != 0); if (!McpServer().Start(mcpBind, mcpPort)) { Mprintf("McpServer start failed on %s:%d\n", mcpBind.c_str(), mcpPort); } else { @@ -5166,6 +5168,11 @@ BOOL CALLBACK CMy2015RemoteDlg::OfflineProc(CONTEXT_OBJECT* ContextObject) McpServer().OnTerminalClosed(ContextObject); } + // MCP 远程控制的屏幕子连接断开：同步清理会话，避免悬空 subCtx 被连接池复用误路由。 + if (McpServer().IsRunning() && McpServer().IsScreenCtrlContext(ContextObject)) { + McpServer().OnScreenControlClosed(ContextObject); + } + SOCKET nSocket = ContextObject->sClientSocket; CDialogBase* p = (CDialogBase*)ContextObject->hDlg; diff --git a/server/2015Remote/McpServer.cpp b/server/2015Remote/McpServer.cpp index 601eef7..95d1f9c 100644 --- a/server/2015Remote/McpServer.cpp +++ b/server/2015Remote/McpServer.cpp @@ -4,6 +4,8 @@ #include "HostJson.h" // BuildHostJson（单台主机序列化公共函数） #include "context.h" // context 接口 #include "2015RemoteDlg.h" // CMy2015RemoteDlg 成员（m_HostList/m_cs/m_ClientMap）+ VERSION_STR +#include "WebService.h" // 远程控制复用屏幕子连接（StartRemoteDesktop / GetScreenContext / GetScreenSize） +#include "Server.h" // CONTEXT_OBJECT 定义（GetScreenContext 返回 CONTEXT_OBJECT* → context* 上转型） #include "LangManager.h" // _TR（审计日志标题语言映射） #include @@ -1125,6 +1127,14 @@ std::string BuildTerminalOpen(const Json::Value& id, const Json::Value& args, CM std::string BuildTerminalExec(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); std::string BuildTerminalClose(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); +// ===== P5 前置声明（定义见下方「tools/call 分派」前）===== +Json::Value BuildRemoteOpenInputSchema(); +Json::Value BuildRemoteOpenOutputSchema(); +Json::Value BuildRemoteCloseInputSchema(); +Json::Value BuildRemoteCloseOutputSchema(); +std::string BuildRemoteOpen(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); +std::string BuildRemoteClose(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); + // tools/list std::string BuildToolsListResult(const Json::Value& id) { Json::Value result(Json::objectValue); @@ -1328,6 +1338,27 @@ std::string BuildToolsListResult(const Json::Value& id) { } } + // 15) remote_open / remote_close（P5：MCP 远程控制，仅 Windows， + // 安全门：McpRemoteControl=1 且 McpReadonly=0；全程审计 + idle 回收） + if (CMcpServer::Instance().IsRemoteControlEnabled() && !CMcpServer::Instance().IsReadonly()) { + { + Json::Value tool(Json::objectValue); + tool["name"] = "remote_open"; + tool["description"] = u8"在指定在线 Windows 主机上建立远程控制会话（隐藏屏幕子连接）并返回 session_id 与物理屏幕分辨率（screen_w/screen_h）。后续 remote_mouse/remote_keyboard 用归一化坐标（0..1）注入输入，用毕须 remote_close。"; + tool["inputSchema"] = BuildRemoteOpenInputSchema(); + tool["outputSchema"] = BuildRemoteOpenOutputSchema(); + tools.append(tool); + } + { + Json::Value tool(Json::objectValue); + tool["name"] = "remote_close"; + tool["description"] = u8"关闭并释放指定远程控制会话（幂等：会话已不存在也返回成功）。"; + tool["inputSchema"] = BuildRemoteCloseInputSchema(); + tool["outputSchema"] = BuildRemoteCloseOutputSchema(); + tools.append(tool); + } + } + result["tools"] = tools; return BuildResult(id, result); } @@ -2533,6 +2564,287 @@ std::string BuildTerminalClose(const Json::Value& id, const Json::Value& args, C return BuildResult(id, result); } +// ===== P5：MCP 远程控制（remote_open / remote_close）===== + +static const int kScreenCtrlIdleTimeoutSec = 300; // 远程控制 idle 回收超时（秒） + +// 共享前置：校验 McpRemoteControl && !readonly → 解析 id → FindMainContext → 拒绝 LNX/MAC。 +// 成功返回 true 并输出 devId/ctx；失败时 errJson 已写入对应 JSON 错误串。 +static bool ResolveScreenCtrlSessionHost(const Json::Value& id, const Json::Value& args, + CMy2015RemoteDlg* parent, + uint64_t& devId, context*& ctx, std::string& errJson) { + CMcpServer& mcp = CMcpServer::Instance(); + if (!mcp.IsRemoteControlEnabled() || mcp.IsReadonly()) { + errJson = BuildError(id, -32006, + "Remote control is disabled: requires McpRemoteControl=1 and McpReadonly=0"); + return false; + } + + std::string err; + if (!ParseHostIdArg(args, devId, err)) { + errJson = BuildError(id, -32602, err); + return false; + } + + ctx = FindMainContext(parent, devId); + if (!ctx) { + errJson = BuildError(id, -32002, "Host not found or offline: " + std::to_string(devId)); + return false; + } + + CString clientType = ctx->GetAdditionalData(RES_CLIENT_TYPE); + if (clientType == "LNX" || clientType == "MAC") { + errJson = BuildError(id, -32005, "Remote control is only supported on Windows hosts"); + return false; + } + return true; +} + +// remote_open 的 inputSchema（id 必填、timeout_ms 可选） +Json::Value BuildRemoteOpenInputSchema() { + Json::Value props(Json::objectValue); + Json::Value idProp(Json::objectValue); + idProp["type"] = "string"; + idProp["description"] = u8"主机 id，取 list_online_hosts / search_hosts 返回的 id 字段"; + props["id"] = idProp; + Json::Value timeoutProp(Json::objectValue); + timeoutProp["type"] = "integer"; + timeoutProp["description"] = u8"等待屏幕子连接建立 + 分辨率到达的超时毫秒数（可选，默认 20000，上限 600000）"; + props["timeout_ms"] = timeoutProp; + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + Json::Value required(Json::arrayValue); + required.append("id"); + schema["required"] = required; + return schema; +} + +Json::Value BuildRemoteOpenOutputSchema() { + Json::Value props(Json::objectValue); + Json::Value sid(Json::objectValue); + sid["type"] = "string"; + sid["description"] = u8"控制会话 token，后续 remote_mouse / remote_keyboard / remote_close 用"; + props["session_id"] = sid; + Json::Value sw(Json::objectValue); + sw["type"] = "integer"; + sw["description"] = u8"物理屏幕宽度（虚拟桌面像素），用于把归一化坐标 0..1 映射为像素"; + props["screen_w"] = sw; + Json::Value sh(Json::objectValue); + sh["type"] = "integer"; + sh["description"] = u8"物理屏幕高度（虚拟桌面像素）"; + props["screen_h"] = sh; + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + Json::Value required(Json::arrayValue); + required.append("session_id"); + required.append("screen_w"); + required.append("screen_h"); + schema["required"] = required; + return schema; +} + +Json::Value BuildRemoteCloseInputSchema() { + Json::Value props(Json::objectValue); + Json::Value idProp(Json::objectValue); + idProp["type"] = "string"; + idProp["description"] = u8"主机 id，取 list_online_hosts / search_hosts 返回的 id 字段"; + props["id"] = idProp; + Json::Value sid(Json::objectValue); + sid["type"] = "string"; + sid["description"] = u8"remote_open 返回的 session_id"; + props["session_id"] = sid; + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + Json::Value required(Json::arrayValue); + required.append("id"); + required.append("session_id"); + schema["required"] = required; + return schema; +} + +Json::Value BuildRemoteCloseOutputSchema() { + Json::Value props(Json::objectValue); + Json::Value closed(Json::objectValue); + closed["type"] = "boolean"; + closed["description"] = u8"恒为 true（幂等）"; + props["closed"] = closed; + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + Json::Value required(Json::arrayValue); + required.append("closed"); + schema["required"] = required; + return schema; +} + +// tools/call：remote_open（建立隐藏屏幕子连接 + 取物理分辨率） +std::string BuildRemoteOpen(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent) { + CMcpServer& mcp = CMcpServer::Instance(); + mcp.SweepIdleScreenCtrl(kScreenCtrlIdleTimeoutSec); + + uint64_t devId = 0; + context* ctx = nullptr; + std::string errJson; + if (!ResolveScreenCtrlSessionHost(id, args, parent, devId, ctx, errJson)) + return errJson; + + // 多显示器门（§7.2 / §12.1）：Observe 抓主屏，Act 注入在虚拟桌面空间，两者仅单显示器时 + // 重合。客户端心跳 RES_RESOLUTION = "N:W*H"（N=显示器数）；N>1 时归一化坐标会系统性 + // 偏移，阶段一直接拒绝（-32008）。无冒号（老客户端格式）则无法判定，放行不拦。 + { + CString res = ctx->GetAdditionalData(RES_RESOLUTION); + int colon = res.Find(':'); + if (colon > 0) { + CString countStr = res.Left(colon); + int monitorCount = atoi(countStr); + if (monitorCount > 1) + return BuildError(id, -32008, + "Multi-monitor hosts are not supported yet (phase 1): monitor count " + + std::to_string(monitorCount)); + } + } + + // 远程控制复用 Web 的隐藏屏幕子连接（COMMAND_SCREEN_SPY → CScreenSpyDlg → RegisterScreenContext）。 + // 该路径依赖 Web 远程服务已启动（WebSvrPort>0，默认开启）；未启动时子连接无法建立/注册。 + if (!WebService().IsRunning()) + return BuildError(id, -32005, + "Remote control requires the Web remote service (WebSvrPort) to be enabled"); + + // 与人类远程桌面观看互斥：已有屏幕子连接（web 观看）→ 拒绝，避免向人类正在看的画面注入。 + if (WebService().HasActiveSession(devId)) + return BuildError(id, -32003, "Device busy: a remote desktop session is already active for this host"); + + int timeoutMs = kMcpToolTimeoutMs; + std::string t = GetStringArg(args, "timeout_ms"); + if (!t.empty() && IsDigits(t)) { + int v = atoi(t.c_str()); + if (v > 0 && v <= 600000) timeoutMs = v; + } + + std::string sessionId = GenerateRandomToken(); + if (!mcp.BeginScreenCtrlOpen(devId, sessionId)) + return BuildError(id, -32003, "Device busy: a remote control session is already active for this host"); + + // 建立隐藏屏幕子连接（发 COMMAND_SCREEN_SPY、标记 web-triggered）。 + if (!WebService().StartRemoteDesktop(devId)) { + mcp.CloseScreenCtrlSession(devId, sessionId); + // StartRemoteDesktop 先置 web-triggered 再 Send2Client，发送失败需一并清理， + // 否则该设备残留 web-triggered 标记、后续人类 web 观看会误走隐藏会话路径。 + WebService().ClearWebTriggered(devId); + return BuildError(id, -32004, "Failed to start remote desktop session"); + } + + // 轮询等待子连接就绪（HasActiveSession）+ 分辨率到达（TOKEN_BITMAPINFO → NotifyResolutionChange）。 + // 屏幕子连接的注册/分辨率由 CScreenSpyDlg 完成，无 McpServer 回调，故用轮询（open 低频）。 + int screenW = 0, screenH = 0; + context* subCtx = nullptr; + auto deadline = std::chrono::steady_clock::now() + std::chrono::milliseconds(timeoutMs); + bool ready = false; + while (std::chrono::steady_clock::now() < deadline) { + if (WebService().HasActiveSession(devId) && WebService().GetScreenSize(devId, screenW, screenH)) { + subCtx = WebService().GetScreenContext(devId); + ready = (subCtx != nullptr); + break; + } + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + + if (!ready) { + mcp.CloseScreenCtrlSession(devId, sessionId); + if (parent) parent->CloseWebRemoteDesktopByClientID(devId); + WebService().ClearWebTriggered(devId); + return BuildError(id, -32001, "Timeout waiting for screen sub-connection / resolution"); + } + + if (!mcp.MarkScreenCtrlReady(devId, sessionId, subCtx, screenW, screenH)) { + // 并发 remote_close 已关掉会话：放弃本次 open（子连接已由 close 关闭）。 + return BuildError(id, -32002, "Remote control session was closed concurrently: " + sessionId); + } + + // 审计：打开会话（不可关闭）。 + if (parent) { + std::string text = "host " + std::to_string(devId) + " remote-open: session " + sessionId + + " (" + std::to_string(screenW) + "x" + std::to_string(screenH) + ")"; + parent->PostMessageA(WM_SHOWERRORMSG, + (WPARAM)new CString(ToAnsi(text, 936).c_str()), + (LPARAM)new CString(_TR("MCP远程控制"))); + } + + Json::Value result(Json::objectValue); + Json::Value structuredContent(Json::objectValue); + structuredContent["session_id"] = sessionId; + structuredContent["screen_w"] = screenW; + structuredContent["screen_h"] = screenH; + result["structuredContent"] = structuredContent; + + Json::Value content(Json::arrayValue); + Json::Value item(Json::objectValue); + item["type"] = "text"; + item["text"] = sessionId; + content.append(item); + result["content"] = content; + result["isError"] = false; + + return BuildResult(id, result); +} + +// tools/call：remote_close（关闭控制会话；幂等，session_id 不匹配报错） +std::string BuildRemoteClose(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent) { + CMcpServer& mcp = CMcpServer::Instance(); + mcp.SweepIdleScreenCtrl(kScreenCtrlIdleTimeoutSec); + + if (!mcp.IsRemoteControlEnabled() || mcp.IsReadonly()) + return BuildError(id, -32006, "Remote control is disabled: requires McpRemoteControl=1 and McpReadonly=0"); + + uint64_t devId = 0; + std::string err; + if (!ParseHostIdArg(args, devId, err)) + return BuildError(id, -32602, err); + + std::string sessionId = GetStringArg(args, "session_id"); + if (sessionId.empty()) + return BuildError(id, -32602, "Missing required parameter: session_id"); + + int r = mcp.CloseScreenCtrlSession(devId, sessionId); + if (r == 2) + return BuildError(id, -32002, "Remote control session_id mismatch: " + sessionId); + + // 仅当确实关闭了 MCP 会话（r==0）时才关闭隐藏屏幕子连接；r==1（幂等，会话已不存在， + // 如已被 idle 回收）时不得误关人类正在观看的 web 会话（CloseWebRemoteDesktopByClientID + // 会关掉该设备任一 IsWebSession 的对话框）。 + if (r == 0) { + // 关闭隐藏屏幕子连接（幂等）：WM_CLOSE → CScreenSpyDlg 析构 → UnregisterScreenContext。 + if (parent) parent->CloseWebRemoteDesktopByClientID(devId); + WebService().ClearWebTriggered(devId); + } + + // 审计：关闭会话（不可关闭）。 + if (parent) { + std::string text = "host " + std::to_string(devId) + " remote-close: session " + sessionId; + parent->PostMessageA(WM_SHOWERRORMSG, + (WPARAM)new CString(ToAnsi(text, 936).c_str()), + (LPARAM)new CString(_TR("MCP远程控制"))); + } + + Json::Value result(Json::objectValue); + Json::Value structuredContent(Json::objectValue); + structuredContent["closed"] = true; + result["structuredContent"] = structuredContent; + + Json::Value content(Json::arrayValue); + Json::Value item(Json::objectValue); + item["type"] = "text"; + item["text"] = std::string(u8"closed"); + content.append(item); + result["content"] = content; + result["isError"] = false; + + return BuildResult(id, result); +} + // tools/call 分派 std::string BuildToolsCall(const Json::Value& root, CMy2015RemoteDlg* parent) { const Json::Value& id = root["id"]; @@ -2561,6 +2873,8 @@ std::string BuildToolsCall(const Json::Value& root, CMy2015RemoteDlg* parent) { if (toolName == "terminal_open") return BuildTerminalOpen(id, args, parent); if (toolName == "terminal_exec") return BuildTerminalExec(id, args, parent); if (toolName == "terminal_close") return BuildTerminalClose(id, args, parent); + if (toolName == "remote_open") return BuildRemoteOpen(id, args, parent); + if (toolName == "remote_close") return BuildRemoteClose(id, args, parent); return BuildError(id, -32602, "Unknown tool: " + (toolName.empty() ? std::string("(empty)") : toolName)); @@ -2842,6 +3156,80 @@ int CMcpServer::SweepIdleTerminals(time_t idleTimeoutSec) { return (int)toCancel.size(); } +// ===== P5：MCP 远程控制（状态机方法）===== + +bool CMcpServer::IsScreenCtrlContext(context* subCtx) { + std::lock_guard lk(m_ScreenCtrlMutex); + return m_ScreenCtrlContextToDevice.find(subCtx) != m_ScreenCtrlContextToDevice.end(); +} + +bool CMcpServer::BeginScreenCtrlOpen(uint64_t device_id, const std::string& sessionId) { + std::lock_guard lk(m_ScreenCtrlMutex); + if (m_ScreenCtrlSessions.find(device_id) != m_ScreenCtrlSessions.end()) return false; // 单设备单会话 + ScreenCtrlSession s; + s.sessionId = sessionId; + s.lastActiveAt = time(nullptr); + m_ScreenCtrlSessions[device_id] = std::move(s); + return true; +} + +bool CMcpServer::MarkScreenCtrlReady(uint64_t device_id, const std::string& sessionId, + context* subCtx, int screenW, int screenH) { + std::lock_guard lk(m_ScreenCtrlMutex); + auto it = m_ScreenCtrlSessions.find(device_id); + if (it == m_ScreenCtrlSessions.end() || it->second.sessionId != sessionId) return false; + ScreenCtrlSession& s = it->second; + s.subCtx = subCtx; + s.started = true; + s.screenW = screenW; + s.screenH = screenH; + s.lastActiveAt = time(nullptr); + m_ScreenCtrlContextToDevice[subCtx] = device_id; + return true; +} + +int CMcpServer::CloseScreenCtrlSession(uint64_t device_id, const std::string& sessionId) { + std::lock_guard lk(m_ScreenCtrlMutex); + auto it = m_ScreenCtrlSessions.find(device_id); + if (it == m_ScreenCtrlSessions.end()) return 1; // 不存在（幂等） + if (it->second.sessionId != sessionId) return 2; // token 不匹配 + if (it->second.subCtx) m_ScreenCtrlContextToDevice.erase(it->second.subCtx); + m_ScreenCtrlSessions.erase(it); + return 0; +} + +int CMcpServer::SweepIdleScreenCtrl(time_t idleTimeoutSec) { + time_t now = time(nullptr); + std::vector toClose; + { + std::lock_guard lk(m_ScreenCtrlMutex); + for (auto it = m_ScreenCtrlSessions.begin(); it != m_ScreenCtrlSessions.end(); ) { + ScreenCtrlSession& s = it->second; + if (difftime(now, s.lastActiveAt) > (double)idleTimeoutSec) { + if (s.subCtx) m_ScreenCtrlContextToDevice.erase(s.subCtx); + toClose.push_back(it->first); + it = m_ScreenCtrlSessions.erase(it); + } else { + ++it; + } + } + } + // 锁外关闭隐藏对话框（WM_CLOSE → CScreenSpyDlg 析构 → UnregisterScreenContext）。 + for (uint64_t devId : toClose) { + if (m_parent) m_parent->CloseWebRemoteDesktopByClientID(devId); + WebService().ClearWebTriggered(devId); + } + return (int)toClose.size(); +} + +void CMcpServer::OnScreenControlClosed(context* subCtx) { + std::lock_guard lk(m_ScreenCtrlMutex); + auto it = m_ScreenCtrlContextToDevice.find(subCtx); + if (it == m_ScreenCtrlContextToDevice.end()) return; + m_ScreenCtrlSessions.erase(it->second); + m_ScreenCtrlContextToDevice.erase(it); +} + ////////////////////////////////////////////////////////////////////////// // CMcpServer Implementation ////////////////////////////////////////////////////////////////////////// diff --git a/server/2015Remote/McpServer.h b/server/2015Remote/McpServer.h index cb15131..928f19c 100644 --- a/server/2015Remote/McpServer.h +++ b/server/2015Remote/McpServer.h @@ -157,6 +157,34 @@ public: // idle 回收：关闭 lastActiveAt 超时且非 busy 的持久会话。返回回收数量。 int SweepIdleTerminals(time_t idleTimeoutSec); + // ===== P5：MCP 远程控制（remote_open / remote_close / remote_mouse / remote_keyboard）===== + // 屏幕子连接由 WebService::StartRemoteDesktop 建立的隐藏 CScreenSpyDlg 持有（复用 Web + // 路径，见 docs/Mcp_RemoteControl_Design.md §6.3）；会话仅记录逻辑状态 + 子连接指针。 + // 注入走该子连接（COMMAND_SCREEN_CONTROL），与终端一样单设备单会话。 + void SetRemoteControlEnabled(bool enabled) { m_remoteControlEnabled = enabled; } + bool IsRemoteControlEnabled() const { return m_remoteControlEnabled; } + + // 该 host 的屏幕子连接是否被 MCP 远程控制会话持有（OfflineProc 反查用）。 + bool IsScreenCtrlContext(context* subCtx); + + // 登记控制会话（单设备单会话；sessionId 由调用方生成）。false = 该设备已有会话。 + bool BeginScreenCtrlOpen(uint64_t device_id, const std::string& sessionId); + + // 子连接就绪（HasActiveSession && 分辨率到达）后调用：填 subCtx/分辨率并置 started。 + // false = 会话已被并发关闭（调用方应放弃本次 open）。 + bool MarkScreenCtrlReady(uint64_t device_id, const std::string& sessionId, + context* subCtx, int screenW, int screenH); + + // 关闭控制会话（校验 sessionId；擦路由 + 会话）。屏幕子连接的关闭由调用方锁外执行 + // （CloseWebRemoteDesktopByClientID）。返回 0=已关；1=不存在（幂等）；2=sessionId 不匹配。 + int CloseScreenCtrlSession(uint64_t device_id, const std::string& sessionId); + + // idle 回收：关闭 lastActiveAt 超时的控制会话，并锁外关闭其隐藏对话框。返回回收数量。 + int SweepIdleScreenCtrl(time_t idleTimeoutSec); + + // 屏幕子连接断开（OfflineProc 调用）：擦除该子连接对应的会话 + 路由（幂等）。 + void OnScreenControlClosed(context* subCtx); + // 安全配置（启动时由 CMy2015RemoteDlg 读 THIS_CFG 后设置）。 void SetReadonly(bool readonly) { m_readonly = readonly; } void SetCmdWhitelist(const std::string& whitelist) { m_cmdWhitelist = whitelist; } @@ -220,9 +248,23 @@ private: std::map m_TermSessions; // device_id → 会话 std::map m_TermContextToDevice; // subCtx → device_id（顶部路由） + // ===== P5：远程控制会话（受 m_ScreenCtrlMutex 保护；单设备单会话）===== + struct ScreenCtrlSession { + std::string sessionId; // 会话 token（每次 open 独立随机） + context* subCtx = nullptr; // 屏幕子连接上下文（就绪后填；用于注入） + bool started = false; // false=子连接建立中；true=已就绪 + time_t lastActiveAt = 0; // idle 回收用（秒） + int screenW = 0; // 物理捕获分辨率（来自 TOKEN_BITMAPINFO） + int screenH = 0; + }; + std::mutex m_ScreenCtrlMutex; + std::map m_ScreenCtrlSessions; // device_id → 会话 + std::map m_ScreenCtrlContextToDevice; // subCtx → device_id（OfflineProc 反查） + bool m_readonly = true; std::string m_cmdWhitelist; - bool m_terminalEnabled = false; // 持久终端开关（默认关；要求 m_readonly=false） + bool m_terminalEnabled = false; // 持久终端开关（默认关；要求 m_readonly=false） + bool m_remoteControlEnabled = false; // 远程控制开关（默认关；要求 m_readonly=false） }; // 全局访问器（仿 WebService()，见 WebService.h 末尾） diff --git a/server/2015Remote/McpSettingsDlg.cpp b/server/2015Remote/McpSettingsDlg.cpp index ffe2427..34bd92b 100644 --- a/server/2015Remote/McpSettingsDlg.cpp +++ b/server/2015Remote/McpSettingsDlg.cpp @@ -106,7 +106,7 @@ INT_PTR CMcpSettingsDlg::DoModal() { USES_CONVERSION; CString title = _TR("MCP设置"); - BuildDialogTemplate(m_Template, T2CW(title), 320, 360); + BuildDialogTemplate(m_Template, T2CW(title), 320, 390); InitModalIndirect((LPCDLGTEMPLATE)m_Template.data()); return CDialog::DoModal(); } @@ -139,6 +139,9 @@ BOOL CMcpSettingsDlg::OnInitDialog() m_btnTerminal.Create(_TR("启用持久终端（全命令，无白名单）"), WS_CHILD | WS_VISIBLE | WS_TABSTOP | BS_AUTOCHECKBOX, r0, this, IDC_MCP_TERMINAL); + m_btnRemoteControl.Create(_TR("启用远程控制（AI 操控桌面）"), + WS_CHILD | WS_VISIBLE | WS_TABSTOP | BS_AUTOCHECKBOX, + r0, this, IDC_MCP_REMOTECONTROL); m_lblWhitelist.Create(_TR("命令白名单"), WS_CHILD | WS_VISIBLE, r0, this, (UINT)-1); m_editWhitelist.Create(WS_CHILD | WS_VISIBLE | WS_BORDER | WS_TABSTOP | ES_MULTILINE | ES_AUTOVSCROLL | ES_WANTRETURN | WS_VSCROLL, @@ -160,6 +163,7 @@ BOOL CMcpSettingsDlg::OnInitDialog() m_editToken.SendMessage(WM_SETFONT, (WPARAM)hFont, MAKELPARAM(TRUE, 0)); m_btnReadonly.SendMessage(WM_SETFONT, (WPARAM)hFont, MAKELPARAM(TRUE, 0)); m_btnTerminal.SendMessage(WM_SETFONT, (WPARAM)hFont, MAKELPARAM(TRUE, 0)); + m_btnRemoteControl.SendMessage(WM_SETFONT, (WPARAM)hFont, MAKELPARAM(TRUE, 0)); m_lblWhitelist.SendMessage(WM_SETFONT, (WPARAM)hFont, MAKELPARAM(TRUE, 0)); m_editWhitelist.SendMessage(WM_SETFONT, (WPARAM)hFont, MAKELPARAM(TRUE, 0)); m_btnOK.SendMessage(WM_SETFONT, (WPARAM)hFont, MAKELPARAM(TRUE, 0)); @@ -174,6 +178,7 @@ BOOL CMcpSettingsDlg::OnInitDialog() if (tok.empty()) tok = GenerateRandomToken(); int readonly = THIS_CFG.GetInt("settings", "McpReadonly", 1); int terminal = THIS_CFG.GetInt("settings", "McpTerminal", 0); + int remoteControl = THIS_CFG.GetInt("settings", "McpRemoteControl", 0); std::string whitelist = THIS_CFG.GetStr("settings", "McpCmdWhitelist", ""); m_btnEnable.SetCheck(enabled ? BST_CHECKED : BST_UNCHECKED); @@ -182,6 +187,7 @@ BOOL CMcpSettingsDlg::OnInitDialog() m_editToken.SetWindowText(CString(tok.c_str())); m_btnReadonly.SetCheck(readonly ? BST_CHECKED : BST_UNCHECKED); m_btnTerminal.SetCheck(terminal ? BST_CHECKED : BST_UNCHECKED); + m_btnRemoteControl.SetCheck(remoteControl ? BST_CHECKED : BST_UNCHECKED); // 白名单存储为逗号分隔，展示为每行一条。 m_editWhitelist.SetWindowText(CString(WhitelistForDisplay(whitelist).c_str())); @@ -199,6 +205,7 @@ void CMcpSettingsDlg::OnOK() bool enabled = (m_btnEnable.GetCheck() == BST_CHECKED); bool readonly = (m_btnReadonly.GetCheck() == BST_CHECKED); bool terminal = (m_btnTerminal.GetCheck() == BST_CHECKED); + bool remoteControl = (m_btnRemoteControl.GetCheck() == BST_CHECKED); // 端口校验：1-65535 int port = atoi(CT2A(sPort)); @@ -223,14 +230,15 @@ void CMcpSettingsDlg::OnOK() THIS_CFG.SetStr("settings", "McpToken", token); THIS_CFG.SetInt("settings", "McpReadonly", readonly ? 1 : 0); THIS_CFG.SetInt("settings", "McpTerminal", terminal ? 1 : 0); + THIS_CFG.SetInt("settings", "McpRemoteControl", remoteControl ? 1 : 0); std::string whitelist = CT2A(sWhitelist); whitelist = NormalizeWhitelist(whitelist); THIS_CFG.SetStr("settings", "McpCmdWhitelist", whitelist); // 拆成两段可翻译的单行键，中间用 \r\n 连接（多行键无法在 INI 中表示） MessageBox(_TR("MCP 设置已保存。") + _T("\r\n") + - _TR("启用/端口/绑定地址/Token/只读/白名单/持久终端的改动需重启程序生效。") + _T("\r\n") + - _TR("持久终端仅在只读模式关闭时生效。"), + _TR("启用/端口/绑定地址/Token/只读/白名单/持久终端/远程控制的改动需重启程序生效。") + _T("\r\n") + + _TR("持久终端与远程控制仅在只读模式关闭时生效。"), _TR("提示"), MB_ICONINFORMATION); CDialog::OnOK(); @@ -267,6 +275,9 @@ void CMcpSettingsDlg::LayoutControls(int cx, int cy) m_btnTerminal.MoveWindow(margin, y, cx - margin * 2, 22); y += 30; + m_btnRemoteControl.MoveWindow(margin, y, cx - margin * 2, 22); + y += 30; + const int whitelistH = 90; m_lblWhitelist.MoveWindow(margin, y, labelW, rowH); m_editWhitelist.MoveWindow(margin + labelW, y - 2, cx - margin * 2 - labelW, whitelistH); diff --git a/server/2015Remote/McpSettingsDlg.h b/server/2015Remote/McpSettingsDlg.h index c11e8aa..54b5623 100644 --- a/server/2015Remote/McpSettingsDlg.h +++ b/server/2015Remote/McpSettingsDlg.h @@ -29,6 +29,7 @@ private: IDC_MCP_READONLY = 1005, // 「只读模式」复选框（默认勾选，禁 exec_command） IDC_MCP_WHITELIST = 1006, // 命令白名单编辑框（多行，逗号/换行分隔，空 = 内置只读前缀） IDC_MCP_TERMINAL = 1007, // 「启用持久终端」复选框（全命令，无白名单，要求只读关） + IDC_MCP_REMOTECONTROL = 1008, // 「启用远程控制」复选框（AI 操控桌面，要求只读关） }; CButton m_btnEnable; @@ -36,6 +37,7 @@ private: CEdit m_editPort, m_editBind, m_editToken; CButton m_btnReadonly; CButton m_btnTerminal; + CButton m_btnRemoteControl; CStatic m_lblWhitelist; CEdit m_editWhitelist; CButton m_btnOK, m_btnCancel; diff --git a/server/2015Remote/WebService.cpp b/server/2015Remote/WebService.cpp index 8b34f09..55c94d0 100644 --- a/server/2015Remote/WebService.cpp +++ b/server/2015Remote/WebService.cpp @@ -1670,6 +1670,17 @@ void CWebService::NotifyResolutionChange(uint64_t device_id, int width, int heig } } +bool CWebService::GetScreenSize(uint64_t device_id, int& width, int& height) { + width = 0; + height = 0; + std::lock_guard lock(m_DeviceCacheMutex); + auto it = m_DeviceCache.find(device_id); + if (it == m_DeviceCache.end()) return false; + width = it->second->screen_width; + height = it->second->screen_height; + return width > 0 && height > 0; +} + void CWebService::NotifyAudioState(uint64_t device_id, bool enabled) { if (m_bStopping) return; diff --git a/server/2015Remote/WebService.h b/server/2015Remote/WebService.h index 41b80be..9cc3f22 100644 --- a/server/2015Remote/WebService.h +++ b/server/2015Remote/WebService.h @@ -104,6 +104,10 @@ public: // Resolution change notification void NotifyResolutionChange(uint64_t device_id, int width, int height, bool top_down = false, const std::string& client_type = ""); + // 读回 NotifyResolutionChange 缓存的物理捕获分辨率（来自 TOKEN_BITMAPINFO）。 + // 尚未收到分辨率时返回 false 并置 w/h 为 0。MCP 远程控制（remote_open）用它映射归一化坐标。 + bool GetScreenSize(uint64_t device_id, int& width, int& height); + // Audio enable/disable notification — pushes current state to all web // clients watching this device and caches it for newcomers. void NotifyAudioState(uint64_t device_id, bool enabled); -- 2.43.0 From 09018e2b4dfd0979e36178ca49a8b79c7d43124b Mon Sep 17 00:00:00 2001 From: yuanyuanxiang <962914132@qq.com> Date: Tue, 25 Aug 2026 13:04:05 +0200 Subject: [PATCH 3/8] Improve: Harden MCP screen-ctrl session state machine for injection Add the busy/closed flags to ScreenCtrlSession plus BeginScreenCtrlAction / EndScreenCtrlAction, mirroring the terminal's busy discipline so an in-flight injection cannot race with session teardown (review finding #6). While an injection is in flight (busy=true), OnScreenControlClosed marks the session closed instead of erasing it (the injection thread, which still holds subCtx, cleans up in EndScreenCtrlAction), SweepIdleScreenCtrl skips it, and CloseScreenCtrlSession defers the erase. The subCtx is looked up under m_ScreenCtrlMutex and the injection is sent outside the lock, matching the established terminal pattern; the screen sub-connection's CONTEXT_OBJECT::Send2Client already serializes internally via SendLock. Co-Authored-By: deepseek-v4-pro --- server/2015Remote/McpServer.cpp | 52 +++++++++++++++++++++++++++++++-- server/2015Remote/McpServer.h | 11 +++++++ 2 files changed, 60 insertions(+), 3 deletions(-) diff --git a/server/2015Remote/McpServer.cpp b/server/2015Remote/McpServer.cpp index 95d1f9c..461c45a 100644 --- a/server/2015Remote/McpServer.cpp +++ b/server/2015Remote/McpServer.cpp @@ -3193,6 +3193,11 @@ int CMcpServer::CloseScreenCtrlSession(uint64_t device_id, const std::string& se auto it = m_ScreenCtrlSessions.find(device_id); if (it == m_ScreenCtrlSessions.end()) return 1; // 不存在（幂等） if (it->second.sessionId != sessionId) return 2; // token 不匹配 + if (it->second.busy) { + // 注入在飞：置 closed 交注入线程收尾，避免与注入并发擦会话（镜像终端 busy 模式）。 + it->second.closed = true; + return 0; + } if (it->second.subCtx) m_ScreenCtrlContextToDevice.erase(it->second.subCtx); m_ScreenCtrlSessions.erase(it); return 0; @@ -3205,7 +3210,7 @@ int CMcpServer::SweepIdleScreenCtrl(time_t idleTimeoutSec) { std::lock_guard lk(m_ScreenCtrlMutex); for (auto it = m_ScreenCtrlSessions.begin(); it != m_ScreenCtrlSessions.end(); ) { ScreenCtrlSession& s = it->second; - if (difftime(now, s.lastActiveAt) > (double)idleTimeoutSec) { + if (!s.busy && difftime(now, s.lastActiveAt) > (double)idleTimeoutSec) { if (s.subCtx) m_ScreenCtrlContextToDevice.erase(s.subCtx); toClose.push_back(it->first); it = m_ScreenCtrlSessions.erase(it); @@ -3226,8 +3231,49 @@ void CMcpServer::OnScreenControlClosed(context* subCtx) { std::lock_guard lk(m_ScreenCtrlMutex); auto it = m_ScreenCtrlContextToDevice.find(subCtx); if (it == m_ScreenCtrlContextToDevice.end()) return; - m_ScreenCtrlSessions.erase(it->second); - m_ScreenCtrlContextToDevice.erase(it); + auto sit = m_ScreenCtrlSessions.find(it->second); + if (sit == m_ScreenCtrlSessions.end()) { // 路由在但会话已擦（防御） + m_ScreenCtrlContextToDevice.erase(it); + return; + } + if (sit->second.busy) { + // 注入在飞：不擦会话（注入线程仍持 subCtx），仅置 closed，由 EndScreenCtrlAction 收尾。 + sit->second.closed = true; + } else { + m_ScreenCtrlSessions.erase(sit); + m_ScreenCtrlContextToDevice.erase(it); + } +} + +int CMcpServer::BeginScreenCtrlAction(uint64_t device_id, const std::string& sessionId, + context*& subCtx, int& screenW, int& screenH) { + std::lock_guard lk(m_ScreenCtrlMutex); + auto it = m_ScreenCtrlSessions.find(device_id); + if (it == m_ScreenCtrlSessions.end()) return 1; // 会话不存在 + ScreenCtrlSession& s = it->second; + if (s.sessionId != sessionId) return 1; // token 不匹配 + if (!s.started) return 1; // 未就绪 + if (s.busy) return 2; // 已有注入在飞 + if (s.closed) return 1; // 子连接已断 + s.busy = true; + s.lastActiveAt = time(nullptr); + subCtx = s.subCtx; + screenW = s.screenW; + screenH = s.screenH; + return 0; +} + +void CMcpServer::EndScreenCtrlAction(uint64_t device_id, const std::string& sessionId) { + std::lock_guard lk(m_ScreenCtrlMutex); + auto it = m_ScreenCtrlSessions.find(device_id); + if (it == m_ScreenCtrlSessions.end()) return; + if (it->second.sessionId != sessionId) return; + it->second.busy = false; + it->second.lastActiveAt = time(nullptr); + if (it->second.closed) { // 注入期间子连接已断：擦会话+路由 + if (it->second.subCtx) m_ScreenCtrlContextToDevice.erase(it->second.subCtx); + m_ScreenCtrlSessions.erase(it); + } } ////////////////////////////////////////////////////////////////////////// diff --git a/server/2015Remote/McpServer.h b/server/2015Remote/McpServer.h index 928f19c..57c27c7 100644 --- a/server/2015Remote/McpServer.h +++ b/server/2015Remote/McpServer.h @@ -175,6 +175,15 @@ public: bool MarkScreenCtrlReady(uint64_t device_id, const std::string& sessionId, context* subCtx, int screenW, int screenH); + // 注入前登记（校验 sessionId/started/!busy/!closed）。输出 subCtx/screenW/H。 + // 注入期间 busy=true，保证 close/sweep/断线不与注入并发擦会话（镜像终端 busy 模式， + // 评审发现 #6）。返回：0=ok；1=不存在/不匹配/未就绪/已断（-32002）；2=忙（-32003）。 + int BeginScreenCtrlAction(uint64_t device_id, const std::string& sessionId, + context*& subCtx, int& screenW, int& screenH); + + // 注入后复位 busy；若注入期间屏幕子连接已断（closed），擦会话+路由（防泄漏）。 + void EndScreenCtrlAction(uint64_t device_id, const std::string& sessionId); + // 关闭控制会话（校验 sessionId；擦路由 + 会话）。屏幕子连接的关闭由调用方锁外执行 // （CloseWebRemoteDesktopByClientID）。返回 0=已关；1=不存在（幂等）；2=sessionId 不匹配。 int CloseScreenCtrlSession(uint64_t device_id, const std::string& sessionId); @@ -253,6 +262,8 @@ private: std::string sessionId; // 会话 token（每次 open 独立随机） context* subCtx = nullptr; // 屏幕子连接上下文（就绪后填；用于注入） bool started = false; // false=子连接建立中；true=已就绪 + bool busy = false; // 一条注入在飞（BeginScreenCtrlAction→End 之间） + bool closed = false; // 注入期间屏幕子连接已断（由注入线程收尾） time_t lastActiveAt = 0; // idle 回收用（秒） int screenW = 0; // 物理捕获分辨率（来自 TOKEN_BITMAPINFO） int screenH = 0; -- 2.43.0 From c844ba7614bfa62a1836608ddc8c492202c31f91 Mon Sep 17 00:00:00 2001 From: yuanyuanxiang <962914132@qq.com> Date: Tue, 25 Aug 2026 13:21:54 +0200 Subject: [PATCH 4/8] Fix: Client misparses screen-control batches when record count is a multiple of 7 ProcessCommand chose the MSG record size by testing "ulLength % 28 == 0" before "% 48 == 0". The two sizes' least common multiple is 336 (7*48 = 12*28), so a 48-byte batch whose record count is a multiple of 7 was misclassified as 28-byte MSG32 records, shifting every field and garbling (or silently dropping) injected input. Check "% 48 == 0" first. The modern controller always emits 48-byte MSG64; the 28-byte MSG32 path is legacy 32-bit-controller compatibility and is only reached when 48 does not divide evenly. This is the first feature (MCP remote_keyboard "type", and the upcoming remote_mouse drag) to emit multi-record batches, which is what made the latent bug reachable. Co-Authored-By: deepseek-v4-pro --- client/ScreenManager.cpp | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/client/ScreenManager.cpp b/client/ScreenManager.cpp index 4247b54..82b7dea 100644 --- a/client/ScreenManager.cpp +++ b/client/ScreenManager.cpp @@ -1726,11 +1726,15 @@ bool IsExtendedKey(WPARAM vKey) VOID CScreenManager::ProcessCommand(LPBYTE szBuffer, ULONG ulLength) { + // 记录大小判定：现代控制端（本服务端）统一发 48 字节 MSG64；28 字节 MSG32 仅为兼容 + // 老 32 位控制端。二者长度的最小公倍数是 336（=7×48=12×28），批量注入（如 MCP 远程 + // 控制的 type/拖拽）时若先判 %28，会把 7 的整数倍条 MSG64 误判成 MSG32，字段错位导致 + // 输入错乱甚至吞掉按键；故先判 %48，仅当不整除 48 才回落到 28。 int msgSize = sizeof(MSG64); - if (ulLength % 28 == 0) // 32位控制端发过来的消息 - msgSize = 28; - else if (ulLength % 48 == 0) // 64位控制端发过来的消息 + if (ulLength % 48 == 0) // 64位控制端（现代服务端）发过来的消息 msgSize = 48; + else if (ulLength % 28 == 0) // 32位控制端发过来的消息（兼容） + msgSize = 28; else return; // 数据包不合法 // 命令个数 -- 2.43.0 From f04d892ac8c6e94c57e9dc419ea7312ae8507f3e Mon Sep 17 00:00:00 2001 From: yuanyuanxiang <962914132@qq.com> Date: Tue, 25 Aug 2026 13:22:00 +0200 Subject: [PATCH 5/8] Feature: Add remote_keyboard MCP tool (key_down / key_up / key_press / type) Implement milestone M2b of the MCP remote-control design: inject keyboard events through an existing screen sub-connection. - Extract the MSG64 keyboard construction from WebService::HandleKey into a shared inline BuildKeyMsg64 helper in WebService.h, and have HandleKey use it (behaviour-preserving) so the Web and MCP paths cannot drift. - Add BuildRemoteKeyboard with four actions: key_down / key_up / key_press (key name -> VK via MapKeyNameToVk, plus CTRL/ALT/SHIFT/WIN modifiers) and type (per-character VkKeyScanA mapping, ASCII only, newline/tab -> Enter/Tab). The batch is sent as one [COMMAND_SCREEN_CONTROL][MSG64*N] packet over the screen sub-connection under the existing Begin/EndScreenCtrlAction busy discipline, then audited via WM_SHOWERRORMSG. Non-ASCII text is rejected (-32602) and must go through remote_clipboard + Ctrl+V (milestone M4), matching the design's clipboard path for CJK input. Co-Authored-By: deepseek-v4-pro --- server/2015Remote/McpServer.cpp | 269 +++++++++++++++++++++++++++++++ server/2015Remote/WebService.cpp | 37 +---- server/2015Remote/WebService.h | 30 ++++ 3 files changed, 301 insertions(+), 35 deletions(-) diff --git a/server/2015Remote/McpServer.cpp b/server/2015Remote/McpServer.cpp index 461c45a..a234ea2 100644 --- a/server/2015Remote/McpServer.cpp +++ b/server/2015Remote/McpServer.cpp @@ -1132,8 +1132,11 @@ Json::Value BuildRemoteOpenInputSchema(); Json::Value BuildRemoteOpenOutputSchema(); Json::Value BuildRemoteCloseInputSchema(); Json::Value BuildRemoteCloseOutputSchema(); +Json::Value BuildRemoteKeyboardInputSchema(); +Json::Value BuildRemoteKeyboardOutputSchema(); std::string BuildRemoteOpen(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); std::string BuildRemoteClose(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); +std::string BuildRemoteKeyboard(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); // tools/list std::string BuildToolsListResult(const Json::Value& id) { @@ -1357,6 +1360,14 @@ std::string BuildToolsListResult(const Json::Value& id) { tool["outputSchema"] = BuildRemoteCloseOutputSchema(); tools.append(tool); } + { + Json::Value tool(Json::objectValue); + tool["name"] = "remote_keyboard"; + tool["description"] = u8"向已建立的远程控制会话注入键盘事件。action 可选 key_down / key_up / key_press / type：前三种需 key（Windows 虚拟键名，如 ENTER/TAB/F5/LEFT/CTRL/ALT/SHIFT/WIN），type 需 text（仅 ASCII 文本；非 ASCII 请用 remote_clipboard + Ctrl+V）。modifiers 可选（CTRL/ALT/SHIFT/WIN）。"; + tool["inputSchema"] = BuildRemoteKeyboardInputSchema(); + tool["outputSchema"] = BuildRemoteKeyboardOutputSchema(); + tools.append(tool); + } } result["tools"] = tools; @@ -2845,6 +2856,263 @@ std::string BuildRemoteClose(const Json::Value& id, const Json::Value& args, CMy return BuildResult(id, result); } +// ===== P5：remote_keyboard（键盘注入，key_down / key_up / key_press / type）===== + +// 键名 → 虚拟键码（对齐 Windows VK_* 去掉前缀，见设计 §4.3）。 +// 支持：单字符 A-Z/0-9、F1-F24、命名键、修饰键。未知返回 0（VK 0 非真实键）。 +static int MapKeyNameToVk(const std::string& nameIn) { + std::string n; + for (char c : nameIn) + n += (c >= 'a' && c <= 'z') ? (char)(c - 'a' + 'A') : c; + + if (n.size() == 1) { + unsigned char c = (unsigned char)n[0]; + if (c >= 'A' && c <= 'Z') return c; // 'A'..'Z' == VK_A..VK_Z + if (c >= '0' && c <= '9') return c; // '0'..'9' == VK_0..VK_9 + return 0; // 标点等请走 type + } + + // F1..F24 + if (n.size() >= 2 && n.size() <= 3 && n[0] == 'F') { + int fnum = atoi(n.c_str() + 1); + if (fnum >= 1 && fnum <= 24) return VK_F1 + (fnum - 1); + return 0; + } + + struct Named { const char* name; int vk; }; + static const Named kNames[] = { + {"ENTER", VK_RETURN}, {"RETURN", VK_RETURN}, {"TAB", VK_TAB}, {"SPACE", VK_SPACE}, + {"ESC", VK_ESCAPE}, {"ESCAPE", VK_ESCAPE}, {"BACKSPACE", VK_BACK}, + {"DELETE", VK_DELETE}, {"DEL", VK_DELETE}, {"INSERT", VK_INSERT}, {"INS", VK_INSERT}, + {"HOME", VK_HOME}, {"END", VK_END}, {"PAGEUP", VK_PRIOR}, {"PRIOR", VK_PRIOR}, + {"PAGEDOWN", VK_NEXT}, {"NEXT", VK_NEXT}, + {"LEFT", VK_LEFT}, {"RIGHT", VK_RIGHT}, {"UP", VK_UP}, {"DOWN", VK_DOWN}, + {"CAPSLOCK", VK_CAPITAL}, {"CAPITAL", VK_CAPITAL}, {"NUMLOCK", VK_NUMLOCK}, + {"SCROLLLOCK", VK_SCROLL}, {"PRINTSCREEN", VK_SNAPSHOT}, {"SNAPSHOT", VK_SNAPSHOT}, + {"PAUSE", VK_PAUSE}, {"APPS", VK_APPS}, {"CONTEXTMENU", VK_APPS}, + {"CTRL", VK_CONTROL}, {"CONTROL", VK_CONTROL}, {"ALT", VK_MENU}, {"MENU", VK_MENU}, + {"SHIFT", VK_SHIFT}, {"WIN", VK_LWIN}, {"WINDOWS", VK_LWIN}, {"CMD", VK_LWIN}, {"SUPER", VK_LWIN}, + }; + for (const Named& e : kNames) + if (n == e.name) return e.vk; + return 0; +} + +// 是否修饰键（modifiers 白名单：CTRL/ALT/SHIFT/WIN） +static bool IsModifierVk(int vk) { + return vk == VK_CONTROL || vk == VK_MENU || vk == VK_SHIFT || vk == VK_LWIN || vk == VK_RWIN; +} + +// 解析可选 modifiers 数组（JSON 字符串数组）→ 修饰键 VK 序列。非法返回 false 并填 err。 +static bool ParseModifiers(const Json::Value& args, std::vector& out, std::string& err) { + out.clear(); + if (!args.isMember("modifiers")) return true; + const Json::Value& mods = args["modifiers"]; + if (mods.isNull()) return true; + if (!mods.isArray()) { err = "modifiers must be an array of strings"; return false; } + for (const auto& m : mods) { + if (out.size() >= 8) { err = "too many modifiers (max 8)"; return false; } + int vk = MapKeyNameToVk(m.asString()); + if (vk == 0 || !IsModifierVk(vk)) { + err = "invalid modifier: " + (m.isString() ? m.asString() : std::string("(non-string)")); + return false; + } + out.push_back(vk); + } + return true; +} + +// 是否纯 ASCII（type 走物理键事件，只能可靠注入 ASCII/ANSI；见设计 §6.4） +static bool IsAscii(const std::string& s) { + for (unsigned char c : s) if (c >= 0x80) return false; + return true; +} + +// type：逐字符 VkKeyScanA 映射为「（Shift↓）字符↓ 字符↑（Shift↑）」，回车/制表符映射为按键。 +// 注：VkKeyScanA 在服务端按服务端线程键盘布局解析字符→虚拟键；仅限 ASCII（0..127）时 +// 各布局的字母/数字/常用标点虚拟键一致，跨布局差异可忽略（非 ASCII 一律走剪贴板）。 +static void BuildTypeBatch(const std::string& text, std::vector& batch) { + for (unsigned char ch : text) { + if (ch == '\r' || ch == '\n') { + batch.push_back(BuildKeyMsg64(VK_RETURN, true, false)); + batch.push_back(BuildKeyMsg64(VK_RETURN, false, false)); + continue; + } + if (ch == '\t') { + batch.push_back(BuildKeyMsg64(VK_TAB, true, false)); + batch.push_back(BuildKeyMsg64(VK_TAB, false, false)); + continue; + } + SHORT scan = VkKeyScanA((char)ch); + if (scan == -1) continue; // 无法映射（控制字符等）→ 跳过 + int vk = scan & 0xFF; + bool needShift = (scan >> 8) & 1; + if (needShift) batch.push_back(BuildKeyMsg64(VK_SHIFT, true, false)); + batch.push_back(BuildKeyMsg64(vk, true, false)); + batch.push_back(BuildKeyMsg64(vk, false, false)); + if (needShift) batch.push_back(BuildKeyMsg64(VK_SHIFT, false, false)); + } +} + +Json::Value BuildRemoteKeyboardInputSchema() { + Json::Value props(Json::objectValue); + Json::Value idProp(Json::objectValue); + idProp["type"] = "string"; + idProp["description"] = u8"主机 id，取 list_online_hosts / search_hosts 返回的 id 字段"; + props["id"] = idProp; + + Json::Value sid(Json::objectValue); + sid["type"] = "string"; + sid["description"] = u8"remote_open 返回的 session_id"; + props["session_id"] = sid; + + Json::Value act(Json::objectValue); + act["type"] = "string"; + act["description"] = u8"动作：key_down / key_up / key_press / type"; + props["action"] = act; + + Json::Value key(Json::objectValue); + key["type"] = "string"; + key["description"] = u8"按键（key_down/key_up/key_press 用）：Windows 虚拟键名（如 ENTER/TAB/F5/LEFT/CTRL/ALT/SHIFT/WIN）或单个字母/数字"; + props["key"] = key; + + Json::Value text(Json::objectValue); + text["type"] = "string"; + text["description"] = u8"文本（type 用，仅 ASCII；非 ASCII 请走 remote_clipboard + Ctrl+V）"; + props["text"] = text; + + Json::Value mods(Json::objectValue); + mods["type"] = "array"; + Json::Value modItem(Json::objectValue); + modItem["type"] = "string"; + mods["items"] = modItem; + mods["description"] = u8"修饰键（可选）：CTRL/ALT/SHIFT/WIN"; + props["modifiers"] = mods; + + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + Json::Value required(Json::arrayValue); + required.append("id"); + required.append("session_id"); + required.append("action"); + schema["required"] = required; + return schema; +} + +Json::Value BuildRemoteKeyboardOutputSchema() { + Json::Value props(Json::objectValue); + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + return schema; +} + +// tools/call：remote_keyboard（键盘注入；成功返回空对象） +std::string BuildRemoteKeyboard(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent) { + CMcpServer& mcp = CMcpServer::Instance(); + mcp.SweepIdleScreenCtrl(kScreenCtrlIdleTimeoutSec); + + uint64_t devId = 0; + context* ctx = nullptr; + std::string errJson; + if (!ResolveScreenCtrlSessionHost(id, args, parent, devId, ctx, errJson)) + return errJson; + + std::string sessionId = GetStringArg(args, "session_id"); + if (sessionId.empty()) + return BuildError(id, -32602, "Missing required parameter: session_id"); + + std::string action = GetStringArg(args, "action"); + if (action.empty()) + return BuildError(id, -32602, "Missing required parameter: action"); + + std::string keyName = GetStringArg(args, "key"); + std::string text = GetStringArg(args, "text"); + std::vector mods; + { + std::string mErr; + if (!ParseModifiers(args, mods, mErr)) + return BuildError(id, -32602, mErr); + } + + // 组装注入批次（按发送顺序） + std::vector batch; + if (action == "type") { + if (text.empty()) + return BuildError(id, -32602, "Missing required parameter: text"); + if (!IsAscii(text)) + return BuildError(id, -32602, "text must be ASCII; non-ASCII text requires remote_clipboard + Ctrl+V"); + if (text.size() > 1024) + return BuildError(id, -32602, "text too long (max 1024 bytes); use remote_clipboard + Ctrl+V for long text"); + BuildTypeBatch(text, batch); + } else if (action == "key_down" || action == "key_up" || action == "key_press") { + if (keyName.empty()) + return BuildError(id, -32602, "Missing required parameter: key"); + int keyVk = MapKeyNameToVk(keyName); + if (keyVk == 0) + return BuildError(id, -32602, "Unknown key: " + keyName); + bool doDown = (action != "key_up"); + bool doUp = (action != "key_down"); + if (doDown) { + for (int m : mods) batch.push_back(BuildKeyMsg64(m, true, false)); + batch.push_back(BuildKeyMsg64(keyVk, true, false)); + } + if (doUp) { + batch.push_back(BuildKeyMsg64(keyVk, false, false)); + for (auto it = mods.rbegin(); it != mods.rend(); ++it) + batch.push_back(BuildKeyMsg64(*it, false, false)); + } + } else { + return BuildError(id, -32602, "Unknown action: " + action); + } + + if (batch.empty()) + return BuildError(id, -32602, "No key events generated"); + + // 校验 session_id/busy 并置 busy（注入期间 close/sweep/断线不擦会话，镜像终端 busy 模式）。 + context* subCtx = nullptr; + int screenW = 0, screenH = 0; + int r = mcp.BeginScreenCtrlAction(devId, sessionId, subCtx, screenW, screenH); + if (r == 2) return BuildError(id, -32003, "Device busy: another injection is in flight for this session"); + if (r == 1) return BuildError(id, -32002, "Remote control session not found, not ready, or closed: " + sessionId); + + // 组包 [COMMAND_SCREEN_CONTROL][MSG64*N]，经屏幕子连接发送（主连接无效，设计 P2）。 + const int len = (int)(1 + batch.size() * sizeof(MSG64)); + std::vector packet((size_t)len); + packet[0] = COMMAND_SCREEN_CONTROL; + memcpy(packet.data() + 1, batch.data(), batch.size() * sizeof(MSG64)); + bool ok = subCtx->Send2Client(packet.data(), (ULONG)len) != FALSE; + + mcp.EndScreenCtrlAction(devId, sessionId); + + if (!ok) + return BuildError(id, -32004, "Failed to send keyboard injection"); + + // 审计（不可关闭） + if (parent) { + std::string audit = "host " + std::to_string(devId) + " remote-keyboard: session " + sessionId + + " action=" + action; + if (action == "type") audit += " text=" + text; + else audit += " key=" + keyName; + parent->PostMessageA(WM_SHOWERRORMSG, + (WPARAM)new CString(ToAnsi(audit, 936).c_str()), + (LPARAM)new CString(_TR("MCP远程控制"))); + } + + Json::Value result(Json::objectValue); + result["structuredContent"] = Json::Value(Json::objectValue); + Json::Value content(Json::arrayValue); + Json::Value item(Json::objectValue); + item["type"] = "text"; + item["text"] = std::string(u8"ok"); + content.append(item); + result["content"] = content; + result["isError"] = false; + + return BuildResult(id, result); +} + // tools/call 分派 std::string BuildToolsCall(const Json::Value& root, CMy2015RemoteDlg* parent) { const Json::Value& id = root["id"]; @@ -2875,6 +3143,7 @@ std::string BuildToolsCall(const Json::Value& root, CMy2015RemoteDlg* parent) { if (toolName == "terminal_close") return BuildTerminalClose(id, args, parent); if (toolName == "remote_open") return BuildRemoteOpen(id, args, parent); if (toolName == "remote_close") return BuildRemoteClose(id, args, parent); + if (toolName == "remote_keyboard") return BuildRemoteKeyboard(id, args, parent); return BuildError(id, -32602, "Unknown tool: " + (toolName.empty() ? std::string("(empty)") : toolName)); diff --git a/server/2015Remote/WebService.cpp b/server/2015Remote/WebService.cpp index 55c94d0..b969159 100644 --- a/server/2015Remote/WebService.cpp +++ b/server/2015Remote/WebService.cpp @@ -907,41 +907,8 @@ void CWebService::HandleKey(void* ws_ptr, const std::string& msg) { return; } - // Build MSG64 structure - MSG64 msg64; - memset(&msg64, 0, sizeof(MSG64)); - - // Use WM_SYSKEYDOWN/UP for Alt combinations (same as MFC version) - if (altKey) { - msg64.message = isDown ? WM_SYSKEYDOWN : WM_SYSKEYUP; - } else { - msg64.message = isDown ? WM_KEYDOWN : WM_KEYUP; - } - msg64.wParam = keyCode; - msg64.time = GetTickCount(); - - // Build lParam for keyboard message: - // bits 0-15: repeat count (1) - // bits 16-23: scan code - // bit 24: extended key flag - // bit 29: context code (1 if Alt is pressed) - // bit 30: previous key state (1 for keyup) - // bit 31: transition state (1 for keyup) - UINT scanCode = MapVirtualKey(keyCode, MAPVK_VK_TO_VSC); - - // Extended keys: arrows, insert, delete, home, end, page up/down, numpad enter, etc. - bool isExtended = (keyCode >= VK_PRIOR && keyCode <= VK_DOWN) || // Page Up/Down, End, Home, Arrows - keyCode == VK_INSERT || keyCode == VK_DELETE || - keyCode == VK_NUMLOCK || keyCode == VK_RCONTROL || keyCode == VK_RMENU || - keyCode == VK_APPS; - - LPARAM lParam = 1; // repeat count = 1 - lParam |= (scanCode & 0xFF) << 16; - if (isExtended) lParam |= (1 << 24); - if (altKey) lParam |= (1 << 29); // context code for Alt - if (!isDown) lParam |= (3UL << 30); // bit 30 and 31 set for key up - - msg64.lParam = lParam; + // Build MSG64 structure（共享 helper，见 WebService.h；与 MCP remote_keyboard 复用） + MSG64 msg64 = BuildKeyMsg64(keyCode, isDown, altKey); // Send command to device const int length = sizeof(MSG64) + 1; diff --git a/server/2015Remote/WebService.h b/server/2015Remote/WebService.h index 9cc3f22..2fdeeed 100644 --- a/server/2015Remote/WebService.h +++ b/server/2015Remote/WebService.h @@ -13,11 +13,41 @@ #include #include +#include "common/commands.h" // MSG64 / COMMAND_SCREEN_CONTROL：共享键盘注入 helper（BuildKeyMsg64） + // Forward declarations class context; class CMy2015RemoteDlg; class CONTEXT_OBJECT; +// 构造单个键盘注入 MSG64（复用 WebService::HandleKey 的 lParam 逻辑，抽取为共享 helper， +// 供 Web 与 MCP 远程控制两处使用，避免漂移，见 docs/Mcp_RemoteControl_Design.md §6.2）。 +// keyCode：虚拟键码；isDown：按下/抬起；isSysKey：true 用 WM_SYSKEYDOWN/UP 并置 Alt 上下文位。 +inline MSG64 BuildKeyMsg64(int keyCode, bool isDown, bool isSysKey) { + MSG64 msg64; + memset(&msg64, 0, sizeof(MSG64)); + + if (isSysKey) msg64.message = isDown ? WM_SYSKEYDOWN : WM_SYSKEYUP; + else msg64.message = isDown ? WM_KEYDOWN : WM_KEYUP; + msg64.wParam = (uint64_t)keyCode; + msg64.time = GetTickCount(); + + UINT scanCode = MapVirtualKey(keyCode, MAPVK_VK_TO_VSC); + bool isExtended = (keyCode >= VK_PRIOR && keyCode <= VK_DOWN) || + keyCode == VK_INSERT || keyCode == VK_DELETE || + keyCode == VK_NUMLOCK || keyCode == VK_RCONTROL || keyCode == VK_RMENU || + keyCode == VK_APPS; + + LPARAM lParam = 1; // repeat count = 1 + lParam |= (scanCode & 0xFF) << 16; + if (isExtended) lParam |= (1 << 24); + if (isSysKey) lParam |= (1 << 29); // context code for Alt + if (!isDown) lParam |= (3UL << 30); // bit 30/31 for key up + + msg64.lParam = (uint64_t)lParam; + return msg64; +} + // Web client state struct WebClient { std::string token; -- 2.43.0 From 18259b8526062db16c8cfddffc78dddc87b7842e Mon Sep 17 00:00:00 2001 From: yuanyuanxiang <962914132@qq.com> Date: Tue, 25 Aug 2026 13:38:09 +0200 Subject: [PATCH 6/8] Feature: Add remote_mouse MCP tool Add the remote_mouse tool to the MCP remote control surface, injecting mouse events (move / down / up / click / right_click / middle_click / drag / scroll) into an established remote_open session. Normalized 0..1 coordinates are mapped to physical pixels via the session's captured resolution and clamped to screen bounds. Scroll follows the existing web console's wheel sign convention (positive delta scrolls down) and is vertical-only. Extract BuildMouseMsg64 into WebService.h as a shared helper, reused by both the web console's HandleMouse and the new MCP handler so the two injection paths cannot drift. Co-Authored-By: deepseek-v4-pro --- server/2015Remote/McpServer.cpp | 296 +++++++++++++++++++++++++++++++ server/2015Remote/WebService.cpp | 41 ++--- server/2015Remote/WebService.h | 15 ++ 3 files changed, 321 insertions(+), 31 deletions(-) diff --git a/server/2015Remote/McpServer.cpp b/server/2015Remote/McpServer.cpp index a234ea2..29d47fd 100644 --- a/server/2015Remote/McpServer.cpp +++ b/server/2015Remote/McpServer.cpp @@ -1134,9 +1134,12 @@ Json::Value BuildRemoteCloseInputSchema(); Json::Value BuildRemoteCloseOutputSchema(); Json::Value BuildRemoteKeyboardInputSchema(); Json::Value BuildRemoteKeyboardOutputSchema(); +Json::Value BuildRemoteMouseInputSchema(); +Json::Value BuildRemoteMouseOutputSchema(); std::string BuildRemoteOpen(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); std::string BuildRemoteClose(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); std::string BuildRemoteKeyboard(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); +std::string BuildRemoteMouse(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); // tools/list std::string BuildToolsListResult(const Json::Value& id) { @@ -1368,6 +1371,14 @@ std::string BuildToolsListResult(const Json::Value& id) { tool["outputSchema"] = BuildRemoteKeyboardOutputSchema(); tools.append(tool); } + { + Json::Value tool(Json::objectValue); + tool["name"] = "remote_mouse"; + tool["description"] = u8"向已建立的远程控制会话注入鼠标事件。action 可选 move / down / up / click / right_click / middle_click / drag / scroll；坐标 x/y（及 drag 的 x2/y2）为归一化 0..1 浮点。button 可选 left/middle/right（默认 left）；click 的 clicks 可选 1/2/3（默认 1）；scroll 的 delta 正=向下滚、负=向上滚。"; + tool["inputSchema"] = BuildRemoteMouseInputSchema(); + tool["outputSchema"] = BuildRemoteMouseOutputSchema(); + tools.append(tool); + } } result["tools"] = tools; @@ -3113,6 +3124,290 @@ std::string BuildRemoteKeyboard(const Json::Value& id, const Json::Value& args, return BuildResult(id, result); } +// ===== P5：remote_mouse（鼠标注入，move/down/up/click/right_click/middle_click/drag/scroll）===== + +// 读取归一化坐标（0..1 浮点，接受 JSON number 或数字字符串），越界钳到 [0,1]。 +static bool GetNormCoord(const Json::Value& args, const char* key, double& out, std::string& err) { + if (!args.isMember(key)) { err = std::string("Missing required parameter: ") + key; return false; } + const Json::Value& v = args[key]; + if (v.isNumeric()) { + out = v.asDouble(); + } else if (v.isString()) { + const std::string s = v.asString(); + if (s.empty()) { err = std::string("Invalid ") + key + ": expected a number in 0..1"; return false; } + char* end = nullptr; + double d = strtod(s.c_str(), &end); + if (end == s.c_str() || *end != '\0') { err = std::string("Invalid ") + key + ": expected a number in 0..1"; return false; } + out = d; + } else { + err = std::string("Invalid ") + key + ": expected a number in 0..1"; + return false; + } + if (out != out) { // NaN（如字符串 "nan"）会绕过钳制并令 (int)(n*screen) 未定义行为 + err = std::string("Invalid ") + key + ": not a finite number"; + return false; + } + if (out < 0.0) out = 0.0; + if (out > 1.0) out = 1.0; + return true; +} + +// 读取可选整数参数（JSON number 或数字字符串）；缺失/非法返回 false（不写 out）。 +static bool GetIntArg(const Json::Value& args, const char* key, int& out) { + if (!args.isMember(key)) return false; + const Json::Value& v = args[key]; + if (v.isNumeric()) { out = v.asInt(); return true; } + if (v.isString()) { + const std::string s = v.asString(); + if (IsDigits(s) || (!s.empty() && s[0] == '-' && IsDigits(s.substr(1)))) { + out = atoi(s.c_str()); + return true; + } + } + return false; +} + +// button 字符串 → 0=左/1=中/2=右；空或 left 默认左。 +static bool ParseButton(const std::string& s, int& out) { + if (s.empty() || s == "left") { out = 0; return true; } + if (s == "middle") { out = 1; return true; } + if (s == "right") { out = 2; return true; } + return false; +} + +// button → 按下/抬起消息 + 按下 wParam（对齐 WebService::HandleMouse）。 +static void ButtonMessages(int button, UINT& downMsg, UINT& upMsg, uint64_t& downWParam) { + if (button == 1) { downMsg = WM_MBUTTONDOWN; upMsg = WM_MBUTTONUP; downWParam = MK_MBUTTON; } + else if (button == 2) { downMsg = WM_RBUTTONDOWN; upMsg = WM_RBUTTONUP; downWParam = MK_RBUTTON; } + else { downMsg = WM_LBUTTONDOWN; upMsg = WM_LBUTTONUP; downWParam = MK_LBUTTON; } +} + +Json::Value BuildRemoteMouseInputSchema() { + Json::Value props(Json::objectValue); + Json::Value idProp(Json::objectValue); + idProp["type"] = "string"; + idProp["description"] = u8"主机 id，取 list_online_hosts / search_hosts 返回的 id 字段"; + props["id"] = idProp; + + Json::Value sid(Json::objectValue); + sid["type"] = "string"; + sid["description"] = u8"remote_open 返回的 session_id"; + props["session_id"] = sid; + + Json::Value act(Json::objectValue); + act["type"] = "string"; + act["description"] = u8"动作：move / down / up / click / right_click / middle_click / drag / scroll"; + Json::Value actEnum(Json::arrayValue); + actEnum.append("move"); actEnum.append("down"); actEnum.append("up"); actEnum.append("click"); + actEnum.append("right_click"); actEnum.append("middle_click"); actEnum.append("drag"); actEnum.append("scroll"); + act["enum"] = actEnum; + props["action"] = act; + + Json::Value xProp(Json::objectValue); + xProp["type"] = "number"; + xProp["description"] = u8"归一化 X 坐标（0=屏幕最左，1=最右）"; + props["x"] = xProp; + + Json::Value yProp(Json::objectValue); + yProp["type"] = "number"; + yProp["description"] = u8"归一化 Y 坐标（0=屏幕最上，1=最下）"; + props["y"] = yProp; + + Json::Value x2Prop(Json::objectValue); + x2Prop["type"] = "number"; + x2Prop["description"] = u8"拖拽终点 X（仅 drag）"; + props["x2"] = x2Prop; + + Json::Value y2Prop(Json::objectValue); + y2Prop["type"] = "number"; + y2Prop["description"] = u8"拖拽终点 Y（仅 drag）"; + props["y2"] = y2Prop; + + Json::Value btn(Json::objectValue); + btn["type"] = "string"; + btn["description"] = u8"按键：left / middle / right（默认 left）"; + Json::Value btnEnum(Json::arrayValue); + btnEnum.append("left"); btnEnum.append("middle"); btnEnum.append("right"); + btn["enum"] = btnEnum; + props["button"] = btn; + + Json::Value clk(Json::objectValue); + clk["type"] = "integer"; + clk["description"] = u8"点击次数 1/2/3（仅 click，默认 1）"; + props["clicks"] = clk; + + Json::Value del(Json::objectValue); + del["type"] = "integer"; + del["description"] = u8"滚动量（仅 scroll；正=向下滚，负=向上滚，与 Web 控制台一致；仅垂直滚轮，忽略 button）"; + props["delta"] = del; + + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + Json::Value required(Json::arrayValue); + required.append("id"); + required.append("session_id"); + required.append("action"); + required.append("x"); + required.append("y"); + schema["required"] = required; + return schema; +} + +Json::Value BuildRemoteMouseOutputSchema() { + Json::Value props(Json::objectValue); + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + return schema; +} + +// tools/call：remote_mouse（鼠标注入；归一化坐标 → 物理像素） +std::string BuildRemoteMouse(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent) { + CMcpServer& mcp = CMcpServer::Instance(); + mcp.SweepIdleScreenCtrl(kScreenCtrlIdleTimeoutSec); + + uint64_t devId = 0; + context* ctx = nullptr; + std::string errJson; + if (!ResolveScreenCtrlSessionHost(id, args, parent, devId, ctx, errJson)) + return errJson; + + std::string sessionId = GetStringArg(args, "session_id"); + if (sessionId.empty()) + return BuildError(id, -32602, "Missing required parameter: session_id"); + + std::string action = GetStringArg(args, "action"); + if (action.empty()) + return BuildError(id, -32602, "Missing required parameter: action"); + + // ---- 纯参数解析（不依赖会话分辨率；BeginScreenCtrlAction 之后再无提前返回）---- + bool knownAction = + action == "move" || action == "down" || action == "up" || action == "click" || + action == "right_click" || action == "middle_click" || action == "drag" || action == "scroll"; + if (!knownAction) + return BuildError(id, -32602, "Unknown action: " + action); + + int button = 0; + std::string buttonStr = GetStringArg(args, "button"); + if (!ParseButton(buttonStr, button)) + return BuildError(id, -32602, "Invalid button: " + buttonStr + " (expected left/middle/right)"); + + int clicks = 1; + if (args.isMember("clicks")) { + if (!GetIntArg(args, "clicks", clicks)) + return BuildError(id, -32602, "clicks must be an integer 1..3"); + if (clicks < 1 || clicks > 3) + return BuildError(id, -32602, "clicks must be 1, 2 or 3"); + } + + int delta = 0; + if (args.isMember("delta") && !GetIntArg(args, "delta", delta)) + return BuildError(id, -32602, "delta must be an integer"); + + int notches = 0; // scroll 实际注入的滚动档数（= clamp(delta, -10, 10)），审计用 + + double nx = 0, ny = 0, nx2 = 0, ny2 = 0; + std::string coordErr; + if (!GetNormCoord(args, "x", nx, coordErr)) return BuildError(id, -32602, coordErr); + if (!GetNormCoord(args, "y", ny, coordErr)) return BuildError(id, -32602, coordErr); + if (action == "drag") { + if (!GetNormCoord(args, "x2", nx2, coordErr)) return BuildError(id, -32602, coordErr); + if (!GetNormCoord(args, "y2", ny2, coordErr)) return BuildError(id, -32602, coordErr); + } + + // 校验 session_id/busy 并置 busy，同时取物理分辨率（screenW/H>0 由会话就绪保证）。 + context* subCtx = nullptr; + int screenW = 0, screenH = 0; + int r = mcp.BeginScreenCtrlAction(devId, sessionId, subCtx, screenW, screenH); + if (r == 2) return BuildError(id, -32003, "Device busy: another injection is in flight for this session"); + if (r == 1) return BuildError(id, -32002, "Remote control session not found, not ready, or closed: " + sessionId); + + // 归一化坐标 → 物理像素（round + clamp 到 [0, screen-1]） + auto toPxX = [&](double n) -> int { int p = (int)(n * screenW + 0.5); return p < 0 ? 0 : (p > screenW - 1 ? screenW - 1 : p); }; + auto toPxY = [&](double n) -> int { int p = (int)(n * screenH + 0.5); return p < 0 ? 0 : (p > screenH - 1 ? screenH - 1 : p); }; + int px = toPxX(nx), py = toPxY(ny); + int px2 = 0, py2 = 0; + + UINT downMsg, upMsg; uint64_t downWParam; + ButtonMessages(button, downMsg, upMsg, downWParam); + + // 组装注入批次（按发送顺序；每个 MSG64 由客户端 ProcessCommand 分发到 SendInput） + std::vector batch; + if (action == "move") { + batch.push_back(BuildMouseMsg64(px, py, WM_MOUSEMOVE, 0)); + } else if (action == "down") { + batch.push_back(BuildMouseMsg64(px, py, downMsg, downWParam)); + } else if (action == "up") { + batch.push_back(BuildMouseMsg64(px, py, upMsg, 0)); + } else if (action == "click" || action == "right_click" || action == "middle_click") { + int b = (action == "right_click") ? 2 : (action == "middle_click" ? 1 : button); + UINT dMsg, uMsg; uint64_t dW; + ButtonMessages(b, dMsg, uMsg, dW); + for (int i = 0; i < clicks; ++i) { + batch.push_back(BuildMouseMsg64(px, py, dMsg, dW)); + batch.push_back(BuildMouseMsg64(px, py, uMsg, 0)); + } + } else if (action == "drag") { + px2 = toPxX(nx2); py2 = toPxY(ny2); + batch.push_back(BuildMouseMsg64(px, py, downMsg, downWParam)); + batch.push_back(BuildMouseMsg64(px2, py2, WM_MOUSEMOVE, 0)); + batch.push_back(BuildMouseMsg64(px2, py2, upMsg, 0)); + } else if (action == "scroll") { + // delta 正=向下滚（与 Web 控制台一致）；WM_MOUSEWHEEL 负值=向下滚，故取反。 + // 客户端对 WM_MOUSEWHEEL 会先 SetCursorPos(x,y) 再 MOUSEEVENTF_WHEEL，单条消息即可定位+滚动。 + notches = delta; + if (notches > 10) notches = 10; + if (notches < -10) notches = -10; + short wheelDelta = (short)(-notches * 120); + batch.push_back(BuildMouseMsg64(px, py, WM_MOUSEWHEEL, MAKEWPARAM(0, wheelDelta))); + } + + // 组包 [COMMAND_SCREEN_CONTROL][MSG64*N]，经屏幕子连接发送（主连接无效，设计 P2）。 + const int len = (int)(1 + batch.size() * sizeof(MSG64)); + std::vector packet((size_t)len); + packet[0] = COMMAND_SCREEN_CONTROL; + memcpy(packet.data() + 1, batch.data(), batch.size() * sizeof(MSG64)); + bool ok = subCtx->Send2Client(packet.data(), (ULONG)len) != FALSE; + + mcp.EndScreenCtrlAction(devId, sessionId); + + if (!ok) + return BuildError(id, -32004, "Failed to send mouse injection"); + + // 审计（不可关闭；含归一化坐标与换算后的物理像素） + if (parent) { + std::string audit = "host " + std::to_string(devId) + " remote-mouse: session " + sessionId + + " action=" + action; + char cbuf[96]; + sprintf(cbuf, " norm=(%.3f,%.3f) px=(%d,%d)", nx, ny, px, py); + audit += cbuf; + if (action == "drag") { + sprintf(cbuf, " -> norm2=(%.3f,%.3f) px2=(%d,%d)", nx2, ny2, px2, py2); + audit += cbuf; + } + if (action == "scroll") + audit += " notches=" + std::to_string(notches); + if (action == "click" || action == "right_click" || action == "middle_click") + audit += " clicks=" + std::to_string(clicks); + parent->PostMessageA(WM_SHOWERRORMSG, + (WPARAM)new CString(ToAnsi(audit, 936).c_str()), + (LPARAM)new CString(_TR("MCP远程控制"))); + } + + Json::Value result(Json::objectValue); + result["structuredContent"] = Json::Value(Json::objectValue); + Json::Value content(Json::arrayValue); + Json::Value item(Json::objectValue); + item["type"] = "text"; + item["text"] = std::string(u8"ok"); + content.append(item); + result["content"] = content; + result["isError"] = false; + + return BuildResult(id, result); +} + // tools/call 分派 std::string BuildToolsCall(const Json::Value& root, CMy2015RemoteDlg* parent) { const Json::Value& id = root["id"]; @@ -3144,6 +3439,7 @@ std::string BuildToolsCall(const Json::Value& root, CMy2015RemoteDlg* parent) { if (toolName == "remote_open") return BuildRemoteOpen(id, args, parent); if (toolName == "remote_close") return BuildRemoteClose(id, args, parent); if (toolName == "remote_keyboard") return BuildRemoteKeyboard(id, args, parent); + if (toolName == "remote_mouse") return BuildRemoteMouse(id, args, parent); return BuildError(id, -32602, "Unknown tool: " + (toolName.empty() ? std::string("(empty)") : toolName)); diff --git a/server/2015Remote/WebService.cpp b/server/2015Remote/WebService.cpp index b969159..2b98056 100644 --- a/server/2015Remote/WebService.cpp +++ b/server/2015Remote/WebService.cpp @@ -799,34 +799,18 @@ void CWebService::HandleMouse(void* ws_ptr, const std::string& msg) { return; } - // Build MSG64 structure - MSG64 msg64; - memset(&msg64, 0, sizeof(MSG64)); - msg64.pt.x = x; - msg64.pt.y = y; - msg64.lParam = MAKELPARAM(x, y); - msg64.time = GetTickCount(); + // Build MSG64 structure（共享 helper，见 WebService.h；与 MCP remote_mouse 复用） + MSG64 msg64 = BuildMouseMsg64(x, y, 0, 0); // 默认 WM_NULL；未知 button 时仍发送（保持原行为） // Map type and button to Windows message if (type == "down") { - if (button == 0) { - msg64.message = WM_LBUTTONDOWN; - msg64.wParam = MK_LBUTTON; - } else if (button == 1) { - msg64.message = WM_MBUTTONDOWN; - msg64.wParam = MK_MBUTTON; - } else if (button == 2) { - msg64.message = WM_RBUTTONDOWN; - msg64.wParam = MK_RBUTTON; - } + if (button == 0) { msg64.message = WM_LBUTTONDOWN; msg64.wParam = MK_LBUTTON; } + else if (button == 1) { msg64.message = WM_MBUTTONDOWN; msg64.wParam = MK_MBUTTON; } + else if (button == 2) { msg64.message = WM_RBUTTONDOWN; msg64.wParam = MK_RBUTTON; } } else if (type == "up") { - if (button == 0) { - msg64.message = WM_LBUTTONUP; - } else if (button == 1) { - msg64.message = WM_MBUTTONUP; - } else if (button == 2) { - msg64.message = WM_RBUTTONUP; - } + if (button == 0) { msg64.message = WM_LBUTTONUP; } + else if (button == 1) { msg64.message = WM_MBUTTONUP; } + else if (button == 2) { msg64.message = WM_RBUTTONUP; } } else if (type == "move") { msg64.message = WM_MOUSEMOVE; } else if (type == "wheel") { @@ -845,13 +829,8 @@ void CWebService::HandleMouse(void* ws_ptr, const std::string& msg) { if (clientType != GetClientType(CLIENT_TYPE_MACOS) && clientType != "macOS") { return; // Skip dblclick for non-macOS clients } - if (button == 0) { - msg64.message = WM_LBUTTONDBLCLK; - msg64.wParam = MK_LBUTTON; - } else if (button == 2) { - msg64.message = WM_RBUTTONDBLCLK; - msg64.wParam = MK_RBUTTON; - } + if (button == 0) { msg64.message = WM_LBUTTONDBLCLK; msg64.wParam = MK_LBUTTON; } + else if (button == 2) { msg64.message = WM_RBUTTONDBLCLK; msg64.wParam = MK_RBUTTON; } } else { return; // Unknown type } diff --git a/server/2015Remote/WebService.h b/server/2015Remote/WebService.h index 2fdeeed..f312822 100644 --- a/server/2015Remote/WebService.h +++ b/server/2015Remote/WebService.h @@ -48,6 +48,21 @@ inline MSG64 BuildKeyMsg64(int keyCode, bool isDown, bool isSysKey) { return msg64; } +// 构造单个鼠标注入 MSG64（复用 WebService::HandleMouse 的基座字段逻辑，抽取为共享 helper， +// 供 Web 与 MCP 远程控制两处使用，见 docs/Mcp_RemoteControl_Design.md §6.2）。 +// x/y 为物理像素坐标（已由调用方从归一化坐标映射）；message/wParam 由调用方按动作设定。 +inline MSG64 BuildMouseMsg64(int x, int y, UINT message, uint64_t wParam) { + MSG64 msg64; + memset(&msg64, 0, sizeof(MSG64)); + msg64.pt.x = x; + msg64.pt.y = y; + msg64.lParam = MAKELPARAM(x, y); + msg64.time = GetTickCount(); + msg64.message = message; + msg64.wParam = wParam; + return msg64; +} + // Web client state struct WebClient { std::string token; -- 2.43.0 From 25a6e2d07a807b435ee99aa80fc5d4d0d143f3e0 Mon Sep 17 00:00:00 2001 From: yuanyuanxiang <962914132@qq.com> Date: Tue, 25 Aug 2026 13:45:51 +0200 Subject: [PATCH 7/8] Feature: Add remote_clipboard MCP tool Add the remote_clipboard tool to the MCP remote control surface, writing text to the remote host's clipboard via COMMAND_SCREEN_SET_CLIPBOARD through the established screen sub-connection. Input UTF-8 is converted to GBK (ToAnsi, code page 936) to match the client's CF_TEXT/ANSI clipboard path, mirroring the existing CScreenSpyDlg::SendServerClipboard packet format. Pasting remains a separate step (remote_keyboard Ctrl+V). Known MVP limitation, documented in the tool description: non-GBK characters (e.g. emoji) are replaced by '?' on the CF_TEXT path. Co-Authored-By: deepseek-v4-pro --- server/2015Remote/McpServer.cpp | 116 ++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) diff --git a/server/2015Remote/McpServer.cpp b/server/2015Remote/McpServer.cpp index 29d47fd..5b7a14b 100644 --- a/server/2015Remote/McpServer.cpp +++ b/server/2015Remote/McpServer.cpp @@ -1136,10 +1136,13 @@ Json::Value BuildRemoteKeyboardInputSchema(); Json::Value BuildRemoteKeyboardOutputSchema(); Json::Value BuildRemoteMouseInputSchema(); Json::Value BuildRemoteMouseOutputSchema(); +Json::Value BuildRemoteClipboardInputSchema(); +Json::Value BuildRemoteClipboardOutputSchema(); std::string BuildRemoteOpen(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); std::string BuildRemoteClose(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); std::string BuildRemoteKeyboard(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); std::string BuildRemoteMouse(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); +std::string BuildRemoteClipboard(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent); // tools/list std::string BuildToolsListResult(const Json::Value& id) { @@ -1379,6 +1382,14 @@ std::string BuildToolsListResult(const Json::Value& id) { tool["outputSchema"] = BuildRemoteMouseOutputSchema(); tools.append(tool); } + { + Json::Value tool(Json::objectValue); + tool["name"] = "remote_clipboard"; + tool["description"] = u8"把文本写入远程主机的剪贴板（UTF-8 → GBK/ANSI，非 GBK 字符如 emoji 会丢失）。仅设置剪贴板，粘贴需随后用 remote_keyboard 注入 Ctrl+V。"; + tool["inputSchema"] = BuildRemoteClipboardInputSchema(); + tool["outputSchema"] = BuildRemoteClipboardOutputSchema(); + tools.append(tool); + } } result["tools"] = tools; @@ -3408,6 +3419,110 @@ std::string BuildRemoteMouse(const Json::Value& id, const Json::Value& args, CMy return BuildResult(id, result); } +// ===== P5：remote_clipboard（写远程剪贴板，UTF-8 → GBK 直发，见设计 §6.4）===== + +Json::Value BuildRemoteClipboardInputSchema() { + Json::Value props(Json::objectValue); + Json::Value idProp(Json::objectValue); + idProp["type"] = "string"; + idProp["description"] = u8"主机 id，取 list_online_hosts / search_hosts 返回的 id 字段"; + props["id"] = idProp; + + Json::Value sid(Json::objectValue); + sid["type"] = "string"; + sid["description"] = u8"remote_open 返回的 session_id"; + props["session_id"] = sid; + + Json::Value text(Json::objectValue); + text["type"] = "string"; + text["description"] = u8"要写入剪贴板的文本（UTF-8；非 GBK 字符如 emoji 会丢失）"; + props["text"] = text; + + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + Json::Value required(Json::arrayValue); + required.append("id"); + required.append("session_id"); + required.append("text"); + schema["required"] = required; + return schema; +} + +Json::Value BuildRemoteClipboardOutputSchema() { + Json::Value props(Json::objectValue); + Json::Value schema(Json::objectValue); + schema["type"] = "object"; + schema["properties"] = props; + return schema; +} + +// tools/call：remote_clipboard（写远程剪贴板；成功返回空对象） +std::string BuildRemoteClipboard(const Json::Value& id, const Json::Value& args, CMy2015RemoteDlg* parent) { + CMcpServer& mcp = CMcpServer::Instance(); + mcp.SweepIdleScreenCtrl(kScreenCtrlIdleTimeoutSec); + + uint64_t devId = 0; + context* ctx = nullptr; + std::string errJson; + if (!ResolveScreenCtrlSessionHost(id, args, parent, devId, ctx, errJson)) + return errJson; + + std::string sessionId = GetStringArg(args, "session_id"); + if (sessionId.empty()) + return BuildError(id, -32602, "Missing required parameter: session_id"); + + std::string text = GetStringArg(args, "text"); + if (text.empty()) + return BuildError(id, -32602, "Missing required parameter: text"); + if (text.size() > 262144) // 256 KB 上限（CF_TEXT 单包，防内存/传输失控） + return BuildError(id, -32602, "text too long (max 262144 bytes)"); + + // UTF-8 → GBK：客户端 UpdateClientClipboard 以 CF_TEXT（ANSI/GBK）落剪贴板（设计 §6.4）。 + // 非 GBK 字符（emoji 等）在此被替换为 '?'，MVP 已知限制。 + std::string ansi = ToAnsi(text, 936); + + // 校验 session_id/busy 并置 busy，取屏幕子连接（注入与剪贴板共用同一子连接串行化）。 + context* subCtx = nullptr; + int screenW = 0, screenH = 0; + int r = mcp.BeginScreenCtrlAction(devId, sessionId, subCtx, screenW, screenH); + if (r == 2) return BuildError(id, -32003, "Device busy: another injection is in flight for this session"); + if (r == 1) return BuildError(id, -32002, "Remote control session not found, not ready, or closed: " + sessionId); + + // 组包 [COMMAND_SCREEN_SET_CLIPBOARD][GBK text]，经屏幕子连接发送（客户端自行补 '\0'）。 + const int len = (int)(1 + ansi.size()); + std::vector packet((size_t)len); + packet[0] = COMMAND_SCREEN_SET_CLIPBOARD; + memcpy(packet.data() + 1, ansi.data(), ansi.size()); + bool ok = subCtx->Send2Client(packet.data(), (ULONG)len) != FALSE; + + mcp.EndScreenCtrlAction(devId, sessionId); + + if (!ok) + return BuildError(id, -32004, "Failed to send clipboard text"); + + // 审计（不可关闭；记字节数，不落全文以免审计日志膨胀） + if (parent) { + std::string audit = "host " + std::to_string(devId) + " remote-clipboard: session " + sessionId + + " input_bytes=" + std::to_string(text.size()); + parent->PostMessageA(WM_SHOWERRORMSG, + (WPARAM)new CString(ToAnsi(audit, 936).c_str()), + (LPARAM)new CString(_TR("MCP远程控制"))); + } + + Json::Value result(Json::objectValue); + result["structuredContent"] = Json::Value(Json::objectValue); + Json::Value content(Json::arrayValue); + Json::Value item(Json::objectValue); + item["type"] = "text"; + item["text"] = std::string(u8"ok"); + content.append(item); + result["content"] = content; + result["isError"] = false; + + return BuildResult(id, result); +} + // tools/call 分派 std::string BuildToolsCall(const Json::Value& root, CMy2015RemoteDlg* parent) { const Json::Value& id = root["id"]; @@ -3440,6 +3555,7 @@ std::string BuildToolsCall(const Json::Value& root, CMy2015RemoteDlg* parent) { if (toolName == "remote_close") return BuildRemoteClose(id, args, parent); if (toolName == "remote_keyboard") return BuildRemoteKeyboard(id, args, parent); if (toolName == "remote_mouse") return BuildRemoteMouse(id, args, parent); + if (toolName == "remote_clipboard") return BuildRemoteClipboard(id, args, parent); return BuildError(id, -32602, "Unknown tool: " + (toolName.empty() ? std::string("(empty)") : toolName)); -- 2.43.0 From d55d40e7a21a10b6a36c9086cf453053bd6c6b73 Mon Sep 17 00:00:00 2001 From: yuanyuanxiang <962914132@qq.com> Date: Tue, 25 Aug 2026 14:14:26 +0200 Subject: [PATCH 8/8] Feature: Exclude human Web viewing from MCP remote control sessions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete the bidirectional mutual exclusion between MCP remote control and human Web viewing (§8.2). Direction 1 (remote_open rejected while a human session holds the screen sub-connection) already existed; this adds direction 2: a human Web viewer is now rejected while an MCP session owns the device. Add a m_McpTriggeredDevices marker (mirroring m_MfcTriggeredDevices) that is set when an MCP session is created (BeginScreenCtrlOpen) and cleared at every session-erasure site (CloseScreenCtrlSession, SweepIdleScreenCtrl, OnScreenControlClosed, EndScreenCtrlAction), so the marker cannot go stale and permanently block humans. HandleConnect checks IsMcpTriggered before mutating client state or starting the remote desktop. Co-Authored-By: deepseek-v4-pro --- server/2015Remote/McpServer.cpp | 12 +++++++++++- server/2015Remote/WebService.cpp | 24 ++++++++++++++++++++++++ server/2015Remote/WebService.h | 10 ++++++++++ 3 files changed, 45 insertions(+), 1 deletion(-) diff --git a/server/2015Remote/McpServer.cpp b/server/2015Remote/McpServer.cpp index 5b7a14b..3000277 100644 --- a/server/2015Remote/McpServer.cpp +++ b/server/2015Remote/McpServer.cpp @@ -3851,6 +3851,10 @@ bool CMcpServer::BeginScreenCtrlOpen(uint64_t device_id, const std::string& sess s.sessionId = sessionId; s.lastActiveAt = time(nullptr); m_ScreenCtrlSessions[device_id] = std::move(s); + // 双向互斥（方向二）：标记该设备屏幕子连接归 MCP 会话独占，人类 Web 观看期间被拒绝 + // （HandleConnect 查 IsMcpTriggered）。会话擦除点（CloseScreenCtrlSession / + // SweepIdleScreenCtrl / OnScreenControlClosed / EndScreenCtrlAction）同步 ClearMcpTriggered。 + WebService().SetMcpTriggered(device_id); return true; } @@ -3881,6 +3885,7 @@ int CMcpServer::CloseScreenCtrlSession(uint64_t device_id, const std::string& se } if (it->second.subCtx) m_ScreenCtrlContextToDevice.erase(it->second.subCtx); m_ScreenCtrlSessions.erase(it); + WebService().ClearMcpTriggered(device_id); return 0; } @@ -3893,8 +3898,10 @@ int CMcpServer::SweepIdleScreenCtrl(time_t idleTimeoutSec) { ScreenCtrlSession& s = it->second; if (!s.busy && difftime(now, s.lastActiveAt) > (double)idleTimeoutSec) { if (s.subCtx) m_ScreenCtrlContextToDevice.erase(s.subCtx); - toClose.push_back(it->first); + uint64_t devId = it->first; + toClose.push_back(devId); it = m_ScreenCtrlSessions.erase(it); + WebService().ClearMcpTriggered(devId); // 锁内清除，避免与并发 remote_open 竞态 } else { ++it; } @@ -3921,8 +3928,10 @@ void CMcpServer::OnScreenControlClosed(context* subCtx) { // 注入在飞：不擦会话（注入线程仍持 subCtx），仅置 closed，由 EndScreenCtrlAction 收尾。 sit->second.closed = true; } else { + uint64_t devId = it->second; m_ScreenCtrlSessions.erase(sit); m_ScreenCtrlContextToDevice.erase(it); + WebService().ClearMcpTriggered(devId); } } @@ -3954,6 +3963,7 @@ void CMcpServer::EndScreenCtrlAction(uint64_t device_id, const std::string& sess if (it->second.closed) { // 注入期间子连接已断：擦会话+路由 if (it->second.subCtx) m_ScreenCtrlContextToDevice.erase(it->second.subCtx); m_ScreenCtrlSessions.erase(it); + WebService().ClearMcpTriggered(device_id); } } diff --git a/server/2015Remote/WebService.cpp b/server/2015Remote/WebService.cpp index 2b98056..bf2777c 100644 --- a/server/2015Remote/WebService.cpp +++ b/server/2015Remote/WebService.cpp @@ -643,6 +643,15 @@ void CWebService::HandleConnect(void* ws_ptr, const std::string& token, uint64_t } } + // Bidirectional mutex (direction 2, §8.2): reject human Web viewing while an MCP + // remote control session owns this host's screen sub-connection (SetMcpTriggered on + // remote_open, cleared on session teardown), so AI input cannot be injected into a + // picture a human is watching/controlling. + if (IsMcpTriggered(device_id)) { + SendText(ws_ptr, BuildJsonResponse("connect_result", false, "Device busy: remote control session active")); + return; + } + // Check max clients per device int current_count = GetWebClientCount(device_id); if (current_count >= m_nMaxClientsPerDevice) { @@ -2150,6 +2159,21 @@ void CWebService::ClearMfcTriggered(uint64_t device_id) { m_MfcTriggeredDevices.erase(device_id); } +void CWebService::SetMcpTriggered(uint64_t device_id) { + std::lock_guard lock(m_McpTriggeredMutex); + m_McpTriggeredDevices.insert(device_id); +} + +bool CWebService::IsMcpTriggered(uint64_t device_id) { + std::lock_guard lock(m_McpTriggeredMutex); + return m_McpTriggeredDevices.find(device_id) != m_McpTriggeredDevices.end(); +} + +void CWebService::ClearMcpTriggered(uint64_t device_id) { + std::lock_guard lock(m_McpTriggeredMutex); + m_McpTriggeredDevices.erase(device_id); +} + bool CWebService::HasActiveSession(uint64_t device_id) { std::lock_guard lock(m_ScreenContextsMutex); return m_ScreenContexts.find(device_id) != m_ScreenContexts.end(); diff --git a/server/2015Remote/WebService.h b/server/2015Remote/WebService.h index f312822..cb4c207 100644 --- a/server/2015Remote/WebService.h +++ b/server/2015Remote/WebService.h @@ -299,6 +299,12 @@ public: bool IsMfcTriggered(uint64_t device_id); void ClearMfcTriggered(uint64_t device_id); + // MCP trigger management - an MCP remote control session owns the screen + // sub-connection and blocks human Web viewing (bidirectional mutex, §8.2). + void SetMcpTriggered(uint64_t device_id); + bool IsMcpTriggered(uint64_t device_id); + void ClearMcpTriggered(uint64_t device_id); + // Check if a remote desktop session already exists for device bool HasActiveSession(uint64_t device_id); @@ -344,6 +350,10 @@ private: std::set m_MfcTriggeredDevices; std::mutex m_MfcTriggeredMutex; + // MCP triggered devices: screen sub-connection owned by an MCP remote control session + std::set m_McpTriggeredDevices; + std::mutex m_McpTriggeredMutex; + // Web 终端会话状态 struct WebTermSession { void* ws_ptr; // browser WebSocket -- 2.43.0