yuanyuanxiang/SimpleRemoter
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 8 Wiki Activity

Feature: MCP 远程控制（remote_open/close/keyboard/mouse/clipboard） #4

Merged
yuanyuanxiang merged 8 commits from feature/mcp-remote-control into main 2026-08-26 12:05:24 +00:00
Conversation 0 Commits 8 Files Changed 10 +1800 -74
yuanyuanxiang commented 2026-08-26 12:04:00 +00:00
Owner
Copy Link

概述

实现 docs/Mcp_RemoteControl_Design.md 的 MCP 远程控制（里程碑 M1–M4）。M5 remote_focus_window 按设计「可后置 / 二选一即可跑通」条款推迟，聚焦用 remote_mouse click 替代。

新增 MCP 工具

工具 说明
remote_open 建立隐藏屏幕子连接 + 取物理分辨率（归一化坐标基准），返回 session_id
remote_close 幂等关闭控制会话
remote_keyboard key_down / key_up / key_press / type（ASCII）
remote_mouse move / down / up / click / right_click / middle_click / drag / scroll（归一化 0..1 → 像素）
remote_clipboard 写远程剪贴板（UTF-8 → GBK → CF_TEXT）

安全模型（可控、可审、可关）

  • 新开关 McpRemoteControl，默认关；要求 McpReadonly=0 且 Web 远程服务已启用。
  • 全程审计（复用 WM_SHOWERRORMSG → get_audit_log）。
  • 单设备单会话；闲置 300s 自动回收；仅 Windows、单显示器（多显示器 -32008）。
  • 双向互斥（§8.2）：人类 Web 观看 ↔ AI 远程控制互斥，避免向人类正在看的画面注入。

实现要点

  • 复用 Web 隐藏屏幕子连接（COMMAND_SCREEN_SPY）；注入走既有 COMMAND_SCREEN_CONTROL（MSG64）/ COMMAND_SCREEN_SET_CLIPBOARD，未新增客户端命令。
  • 键盘/鼠标 MSG64 构造抽为 WebService.h 共享 helper，Web 与 MCP 两处复用，避免漂移。
  • 客户端批量解析顺序修复（%48 优先于 %28），解决 7 的整数倍条 MSG64 被误判为 MSG32 的字段错位。

测试

  • 端到端：remote_open → mouse/keyboard → close 全链路（运行实例 127.0.0.1:6544）。
  • 双向互斥双向验证：方向二（MCP 阻塞人类 Web）即时生效；方向一（人类 Web 阻塞 MCP）子连接注册后生效（约 1s）。
  • 多显示器主机返回 -32008。
## 概述 实现 docs/Mcp_RemoteControl_Design.md 的 MCP 远程控制（里程碑 M1–M4）。M5 remote_focus_window 按设计「可后置 / 二选一即可跑通」条款推迟，聚焦用 remote_mouse click 替代。 ## 新增 MCP 工具 | 工具 | 说明 | |---|---| | remote_open | 建立隐藏屏幕子连接 + 取物理分辨率（归一化坐标基准），返回 session_id | | remote_close | 幂等关闭控制会话 | | remote_keyboard | key_down / key_up / key_press / type（ASCII） | | remote_mouse | move / down / up / click / right_click / middle_click / drag / scroll（归一化 0..1 → 像素） | | remote_clipboard | 写远程剪贴板（UTF-8 → GBK → CF_TEXT） | ## 安全模型（可控、可审、可关） - 新开关 McpRemoteControl，默认关；要求 McpReadonly=0 且 Web 远程服务已启用。 - 全程审计（复用 WM_SHOWERRORMSG → get_audit_log）。 - 单设备单会话；闲置 300s 自动回收；仅 Windows、单显示器（多显示器 -32008）。 - 双向互斥（§8.2）：人类 Web 观看 ↔ AI 远程控制互斥，避免向人类正在看的画面注入。 ## 实现要点 - 复用 Web 隐藏屏幕子连接（COMMAND_SCREEN_SPY）；注入走既有 COMMAND_SCREEN_CONTROL（MSG64）/ COMMAND_SCREEN_SET_CLIPBOARD，未新增客户端命令。 - 键盘/鼠标 MSG64 构造抽为 WebService.h 共享 helper，Web 与 MCP 两处复用，避免漂移。 - 客户端批量解析顺序修复（%48 优先于 %28），解决 7 的整数倍条 MSG64 被误判为 MSG32 的字段错位。 ## 测试 - 端到端：remote_open → mouse/keyboard → close 全链路（运行实例 127.0.0.1:6544）。 - 双向互斥双向验证：方向二（MCP 阻塞人类 Web）即时生效；方向一（人类 Web 阻塞 MCP）子连接注册后生效（约 1s）。 - 多显示器主机返回 -32008。
yuanyuanxiang added 8 commits 2026-08-26 12:04:01 +00:00
doc: Add remote control MCP design doc 84e3565de1 
Add docs/Mcp_RemoteControl_Design.md, the reference design for an
screenshot-driven AI remote-control feature: the existing get_screenshot for
observation plus remote_open/remote_close/remote_mouse/remote_keyboard for
input injection, reusing COMMAND_SCREEN_PREVIEW_REQ and COMMAND_SCREEN_CONTROL
+ MSG64. Coordinates are normalized (0..1) and mapped server-side to physical
pixels; injection runs over the screen sub-connection opened by
WebService::StartRemoteDesktop's hidden CScreenSpyDlg. Includes a
chat-on-behalf experiment case with clipboard-encoding and window-capture
notes.

Also fold in the get_audit_log encoding correction for
docs/Mcp_Terminal_Design.md, so the two doc changes ship as one commit.

Co-Authored-By: deepseek-v4-pro
Feature: Add MCP remote_open/remote_close remote control sessions 6045baadb8 
Add M1 of MCP remote control (docs/Mcp_RemoteControl_Design.md): the
remote_open / remote_close tools plus the ScreenCtrlSession state machine.

remote_open establishes a hidden screen sub-connection by reusing
WebService::StartRemoteDesktop (COMMAND_SCREEN_SPY -> CScreenSpyDlg ->
RegisterScreenContext), polls for the sub-connection plus its physical
resolution (TOKEN_BITMAPINFO -> NotifyResolutionChange -> GetScreenSize),
then records the session (single device, single session, reverse-mapped
subCtx for OfflineProc cleanup) and returns {session_id, screen_w,
screen_h}. remote_close validates session_id and tears down the
sub-connection idempotently. A McpRemoteControl settings checkbox (default
off, requires McpReadonly=0) gates the tools; every open/close is audited
via WM_SHOWERRORMSG.

Gating: multi-monitor hosts are rejected with -32008 (phase 1 supports only
single monitor, where Observe=main screen and Act=virtual desktop coincide);
the monitor count comes from the client heartbeat RES_RESOLUTION ("N:W*H").

Known limitations (deferred to the injection milestones): mutual exclusion
with human remote-desktop viewing is one-directional in M1 (a human who
joins during an MCP session can tear it down on disconnect), and subCtx is
not yet dereferenced so no liveness re-check is needed until
remote_mouse/remote_keyboard.

Co-Authored-By: deepseek-v4-pro
Improve: Harden MCP screen-ctrl session state machine for injection 09018e2b4d 
Add the busy/closed flags to ScreenCtrlSession plus BeginScreenCtrlAction
/ EndScreenCtrlAction, mirroring the terminal's busy discipline so an
in-flight injection cannot race with session teardown (review finding #6).

While an injection is in flight (busy=true), OnScreenControlClosed marks the
session closed instead of erasing it (the injection thread, which still holds
subCtx, cleans up in EndScreenCtrlAction), SweepIdleScreenCtrl skips it, and
CloseScreenCtrlSession defers the erase. The subCtx is looked up under
m_ScreenCtrlMutex and the injection is sent outside the lock, matching the
established terminal pattern; the screen sub-connection's CONTEXT_OBJECT::Send2Client
already serializes internally via SendLock.

Co-Authored-By: deepseek-v4-pro
Fix: Client misparses screen-control batches when record count is a multiple of 7 c844ba7614 
ProcessCommand chose the MSG record size by testing "ulLength % 28 == 0"
before "% 48 == 0". The two sizes' least common multiple is 336 (7*48 =
12*28), so a 48-byte batch whose record count is a multiple of 7 was
misclassified as 28-byte MSG32 records, shifting every field and garbling
(or silently dropping) injected input.

Check "% 48 == 0" first. The modern controller always emits 48-byte MSG64;
the 28-byte MSG32 path is legacy 32-bit-controller compatibility and is only
reached when 48 does not divide evenly. This is the first feature (MCP
remote_keyboard "type", and the upcoming remote_mouse drag) to emit
multi-record batches, which is what made the latent bug reachable.

Co-Authored-By: deepseek-v4-pro
Feature: Add remote_keyboard MCP tool (key_down / key_up / key_press / type) f04d892ac8 
Implement milestone M2b of the MCP remote-control design: inject keyboard
events through an existing screen sub-connection.

- Extract the MSG64 keyboard construction from WebService::HandleKey into a
  shared inline BuildKeyMsg64 helper in WebService.h, and have HandleKey use
  it (behaviour-preserving) so the Web and MCP paths cannot drift.
- Add BuildRemoteKeyboard with four actions: key_down / key_up / key_press
  (key name -> VK via MapKeyNameToVk, plus CTRL/ALT/SHIFT/WIN modifiers) and
  type (per-character VkKeyScanA mapping, ASCII only, newline/tab -> Enter/Tab).
  The batch is sent as one [COMMAND_SCREEN_CONTROL][MSG64*N] packet over the
  screen sub-connection under the existing Begin/EndScreenCtrlAction busy
  discipline, then audited via WM_SHOWERRORMSG.

Non-ASCII text is rejected (-32602) and must go through remote_clipboard +
Ctrl+V (milestone M4), matching the design's clipboard path for CJK input.

Co-Authored-By: deepseek-v4-pro
Feature: Add remote_mouse MCP tool 18259b8526 
Add the remote_mouse tool to the MCP remote control surface, injecting
mouse events (move / down / up / click / right_click / middle_click /
drag / scroll) into an established remote_open session. Normalized 0..1
coordinates are mapped to physical pixels via the session's captured
resolution and clamped to screen bounds. Scroll follows the existing web
console's wheel sign convention (positive delta scrolls down) and is
vertical-only.

Extract BuildMouseMsg64 into WebService.h as a shared helper, reused by
both the web console's HandleMouse and the new MCP handler so the two
injection paths cannot drift.

Co-Authored-By: deepseek-v4-pro
Feature: Add remote_clipboard MCP tool 25a6e2d07a 
Add the remote_clipboard tool to the MCP remote control surface, writing
text to the remote host's clipboard via COMMAND_SCREEN_SET_CLIPBOARD
through the established screen sub-connection. Input UTF-8 is converted
to GBK (ToAnsi, code page 936) to match the client's CF_TEXT/ANSI
clipboard path, mirroring the existing CScreenSpyDlg::SendServerClipboard
packet format. Pasting remains a separate step (remote_keyboard Ctrl+V).

Known MVP limitation, documented in the tool description: non-GBK
characters (e.g. emoji) are replaced by '?' on the CF_TEXT path.

Co-Authored-By: deepseek-v4-pro
Feature: Exclude human Web viewing from MCP remote control sessions d55d40e7a2 
Complete the bidirectional mutual exclusion between MCP remote control
and human Web viewing (§8.2). Direction 1 (remote_open rejected while a
human session holds the screen sub-connection) already existed; this adds
direction 2: a human Web viewer is now rejected while an MCP session owns
the device.

Add a m_McpTriggeredDevices marker (mirroring m_MfcTriggeredDevices) that
is set when an MCP session is created (BeginScreenCtrlOpen) and cleared at
every session-erasure site (CloseScreenCtrlSession, SweepIdleScreenCtrl,
OnScreenControlClosed, EndScreenCtrlAction), so the marker cannot go stale
and permanently block humans. HandleConnect checks IsMcpTriggered before
mutating client state or starting the remote desktop.

Co-Authored-By: deepseek-v4-pro
yuanyuanxiang merged commit d55d40e7a2 into main 2026-08-26 12:05:24 +00:00
yuanyuanxiang deleted branch feature/mcp-remote-control 2026-08-26 12:05:24 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yuanyuanxiang
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: yuanyuanxiang/SimpleRemoter#4
Delete Branch "feature/mcp-remote-control"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?