MCP remote control needs screenshots and mouse injection to share one coordinate space. Previously the client captured only the primary monitor while injection used virtual-desktop coordinates, so multi-monitor hosts were rejected outright.
Client: COMMAND_SCREEN_SPY param buffer[11] (wire index 12) now forces whole-virtual-desktop GDI capture; screen preview JPEGs also cover the virtual desktop, including monitors at negative origins.
Server: advertise CLIENT_CAP_VIRTUAL_DESKTOP, gate multi-monitor remote control on that capability, and pass the flag when starting the session. Linux stays exempt (X root capture/injection are already consistent).
Notes
New clients are required for Windows multi-monitor; old clients still get -32005.
The forced flag does not participate in the human monitor-rotation index.
Test plan
Client and server sources compile
Deploy new client, verify capability bit and dual-monitor capture
Verify MCP remote control coordinates map correctly on a dual-monitor host
## Summary
MCP remote control needs screenshots and mouse injection to share one coordinate space. Previously the client captured only the primary monitor while injection used virtual-desktop coordinates, so multi-monitor hosts were rejected outright.
- **Client**: `COMMAND_SCREEN_SPY` param `buffer[11]` (wire index 12) now forces whole-virtual-desktop GDI capture; screen preview JPEGs also cover the virtual desktop, including monitors at negative origins.
- **Server**: advertise `CLIENT_CAP_VIRTUAL_DESKTOP`, gate multi-monitor remote control on that capability, and pass the flag when starting the session. Linux stays exempt (X root capture/injection are already consistent).
## Notes
- New clients are required for Windows multi-monitor; old clients still get -32005.
- The forced flag does not participate in the human monitor-rotation index.
## Test plan
- [x] Client and server sources compile
- [ ] Deploy new client, verify capability bit and dual-monitor capture
- [ ] Verify MCP remote control coordinates map correctly on a dual-monitor host
The MCP Observe/Act loop requires screenshots and mouse injection to share
one coordinate space. Previously the client sent only the primary monitor
while injection used virtual-desktop coordinates, so multi-monitor hosts
were rejected outright.
- Client: COMMAND_SCREEN_SPY param buffer[11] (wire index 12) now forces
whole-virtual-desktop GDI capture; preview JPEGs also cover the virtual
desktop including negative origins.
- Server: advertise CLIENT_CAP_VIRTUAL_DESKTOP, gate multi-monitor remote
control on that capability, and pass the flag when starting the session.
Linux stays exempt (X root capture/injection are already consistent).
「Linux 天然同源」对传统多 X screen(Zaphod,:0.0 / :0.1)不成立——XDefaultScreen 只取一个 screen。现代 RandR/Xinerama 不受影响;若想严谨,可在 remote_open 描述里限定「单 X screen」。
Test plan 里「Deploy new client, verify capability bit and dual-monitor capture」和「Verify MCP remote control coordinates」两项还未勾选,合并前建议实机跑一遍——尤其是负原点(副屏在主屏左侧/上方)的点击落点。
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
MCP remote control needs screenshots and mouse injection to share one coordinate space. Previously the client captured only the primary monitor while injection used virtual-desktop coordinates, so multi-monitor hosts were rejected outright.
COMMAND_SCREEN_SPYparambuffer[11](wire index 12) now forces whole-virtual-desktop GDI capture; screen preview JPEGs also cover the virtual desktop, including monitors at negative origins.CLIENT_CAP_VIRTUAL_DESKTOP, gate multi-monitor remote control on that capability, and pass the flag when starting the session. Linux stays exempt (X root capture/injection are already consistent).Notes
Test plan
审查结论:无 blocking 问题,建议合并(待 Test plan 两项部署验证完成)
设计方向正确,「Observe / Act 共用同一坐标系」是多显示器远程控制的正确解法。特别确认几处关键实现:
bToken[12]↔ 客户端param->buffer[11](KernelManager.cpp:1604剥掉命令字节,buffer[i] == bToken[i+1]),且明确避开了wire [4..11]的 HWND 字段。注释把映射关系写进代码,避免再出现「服务端写 bToken[11]、客户端读 buffer[11]」那种错一位的坑。能力位 0x0080 无冲突,老客户端不声明 → 不置位 → 仍走 -32005,兼容路径完整。PointConversion第 1 步还原缩放、第 3 步加m_iScreenX/Y(可为负原点),整虚拟桌面的注入换算闭合;CommitMonitorIndex跳过 forceVirtual,不污染人类屏轮转;forceVirtual 强制 GDI 绕开 DXGI 按输出抓取的限制,判断正确;m_ptrUser一直指向 UserParam,质量档/分辨率触发的自适应重启不会丢 buffer[11]。linux/ScreenHandler.h:375用XDisplayWidth(DefaultScreen)+XGetImage(root, 0,0,w,h),m_bmpHeader.biWidth/Height直接等于 root 尺寸、发送帧不缩放,注入侧XTestFakeMotionEvent(..., -1, x, y, 0)用 root 绝对坐标,单 X screen(Xinerama/RandR)下确实天然同源。非阻塞建议(见 inline 评论)
ScreenPreview.cpp改成整虚拟桌面后只按宽度缩放,缺高度上限;Linux 端已有 1080 上限的先例,建议对齐。StartRemoteDesktop的会话复用早退分支静默忽略新增的forceVirtualDesktop参数,正常路径被BuildRemoteOpen的前置门控挡住,但存在竞态窗口。!IsWindowsClient(ctx)豁免范围比caps["remoteControl"]宽,APK 上两者结论不一致;建议收窄成显式clientType == "LNX"。GetAllMonitors()返回空时 forceVirtual 静默失效加兜底日志。两点提醒(不阻塞)
ScreenCapture的 8 个m_BlockBuffers按W*H*8合计约 100 MB(单屏 4K 约 66 MB),H.264 编码分辨率也涨 1.5 倍。MCP 低频调用可接受,但建议在双屏实机上确认CPU/内存。:0.0/:0.1)不成立——XDefaultScreen只取一个 screen。现代 RandR/Xinerama 不受影响;若想严谨,可在remote_open描述里限定「单 X screen」。Test plan 里「Deploy new client, verify capability bit and dual-monitor capture」和「Verify MCP remote control coordinates」两项还未勾选,合并前建议实机跑一遍——尤其是负原点(副屏在主屏左侧/上方)的点击落点。
@@ -122,5 +122,5 @@if (srcW <= 0 || srcH <= 0) return SCREEN_PREVIEW_CAPTURE_FAILED;// 等比缩放,禁止放大int targetW = (srcW <= maxWidth) ? srcW : maxWidth;int targetH = (int)((double)srcH * targetW / srcW + 0.5);建议补高度上限(P2)
改成整虚拟桌面后,缩放只按
maxWidth做等比换算,targetH没有上限。纵向多屏(例如两块 1920×3240 上下排列)或竖屏 4K 时,srcW=1920 <= maxWidth不触发缩放,会直接产出 1920×3240 的 Bitmap 并编码 JPEG,再经get_screenshotbase64 返回——内存和响应体都按像素数线性膨胀。Linux 端
linux/ScreenPreview.cpp:129-142其实已经有对应的处理,注释写得很清楚:「再把高度限制在 1080,避免竖屏 4K 这类极端宽高比把 BGRX 包撑到几十 MB(服务端只按宽度提要求,不管比例)」。建议 Windows 端直接对齐:再做偶数对齐,并保持
targetW <= srcW、targetH <= srcH的不放大约束。顺带一提ScreenPreviewRspHeader.width/height是uint16_t,虽然这个量级不会溢出,但双端尺寸策略一致更好维护。@@ -140,6 +140,7 @@ inline int isValid_10s()#define CLIENT_CAP_C2C_RECV_ACK 0x0010 // 作为 C2C 目标端时,落盘完成后回执(TOKEN_C2C_RECV_RESULT,目前仅 Linux)#define CLIENT_CAP_PROTO_SWITCH 0x0020 // 支持服务端远程切换传输协议(COMMAND_SET_PROTO,仅 Windows)#define CLIENT_CAP_EMBEDDED_ANCHOR 0x0040 // 当前靠「生成时内嵌的 TLS 信任锚」在加密(按运行时状态置位,不是能力声明)#define CLIENT_CAP_VIRTUAL_DESKTOP 0x0080 // 支持按 COMMAND_SCREEN_SPY 参数 buffer[11](wire 索引 12)强制抓取整个虚拟桌面(多显示器远程控制)建议补协议布局测试 + 空 monitors 兜底(P3)
这个能力位刚踩过 wire/param 偏移错一位的坑(服务端写
bToken[11]、客户端读buffer[11],中间差一个命令字节),而本次 11 个改动文件里没有任何测试。建议加一个不依赖 GUI 的小测试或static_assert风格的布局表:构造 32 字节 token → 模拟szBuffer+1剥离 → 断言buffer[11] == 强制标志、buffer[3..10] == HWND。以后再往COMMAND_SCREEN_SPY加字段有护栏。另外
client/ScreenCapture.h的forceVirtual ? 0依赖all && !monitors.empty():若GetAllMonitors()返回空(无显示器/会话异常),会落到单屏else分支,m_iScreenX/Y保持 0,forceVirtual 静默失效但服务端仍认为已生效。建议至少打一条日志,或直接在该分支也设置虚拟桌面几何。@@ -4759,2 +4763,2 @@int monitorCount = atoi(countStr);if (monitorCount > 1)int monitorCount = atoi(res.Left(colon));if (monitorCount > 1 && IsWindowsClient(ctx) && !ctx->SupportsVirtualDesktop())豁免范围建议收窄成显式 LNX(P2)
这里用
IsWindowsClient(ctx)做门,而caps["remoteControl"]里的virtDesktopOk用!IsWindowsClient(ctx)放行。按McpInternal.cpp:176的定义,IsWindowsClient只在 LNX / MAC / APK 时返回 false,于是:ResolveScreenCtrlSessionHost会拒,没问题;caps["remoteControl"]因(IsWindowsClient(ctx) || clientType == "LNX")恒为 false。get_server_status说这台不能做,remote_open却放行,结论不一致。PR 文字说的是「Linux stays exempt」,代码实际豁免的是所有非 Windows。建议两处都改成显式的
clientType == "LNX",并考虑把ResolveScreenCtrlSessionHost的平台门从「只拒 MAC」收紧为「只允许IsWindowsClient(ctx) || clientType == "LNX"」,让 caps 与派发门完全对齐。@@ -2390,3 +2390,3 @@}bool CWebService::StartRemoteDesktop(uint64_t device_id) {bool CWebService::StartRemoteDesktop(uint64_t device_id, bool forceVirtualDesktop) {早退分支会静默忽略新参数(P2)
StartRemoteDesktop前面的复用分支:传
forceVirtualDesktop=true进来时,若命中这里就直接复用了一个没有整虚拟桌面抓屏的旧会话,MCP 后续拿到的 BITMAPINFO 仍是单屏/轮转屏,归一化注入坐标会系统性偏移——而且调用方拿到true,无从察觉。正常路径下
BuildRemoteOpen的HasActiveSession前置拒绝能挡住,但这两次检查不是原子的,存在一个竞态窗口;反方向(MCP 会话存活时人类 Web 观看走WebService.cpp:994的默认false)也会复用整虚拟桌面画面,虽然无害但语义含糊。建议把该标志存进会话状态,复用前比对:一致才 return true,不一致就重建会话或返回明确错误,而不是静默忽略。