Feature: Support remote clipboard and input on Android hosts

The MCP remote-control family (remote_open / remote_close / remote_mouse /
remote_keyboard / remote_clipboard) previously stopped at Windows and Linux,
even though the Android client already translated injected events into touch
gestures and global actions. This wires up the missing piece and opens the five
tools to APK hosts.

Client side, the screen sub-connection now handles COMMAND_SCREEN_SET_CLIPBOARD:
the native handler forwards the raw GBK bytes to CaptureService.onClipboardText,
which decodes them, strips the trailing NUL that the human clipboard-sync
senders append, and writes the system clipboard from a background app (allowed
on Android 10+, only reads are restricted). Pasting cannot be a Ctrl+V combo,
so VK_V is mapped to ACTION_PASTE on the focused node, falling back to the
accessibility focus and then to the first editable node, since WebView and
self-drawn inputs often do not flag FOCUS_INPUT or isEditable.

A new capability bit CLIENT_CAP_SCREEN_CLIPBOARD (0x0800) tells the server the
APK understands both paths, so builds without the handler get an explicit
-32005 instead of a package that is silently dropped. action=type is rejected
outright for APK hosts - physical key events would only turn 'v' into a paste -
and the error points at remote_clipboard + key=V.

The platform table, server instructions and the five tool descriptions now
describe Android support, which keys actually act as global actions (ESC, HOME,
APPS, PRINTSCREEN), and that injection needs the accessibility service while
capture needs an authorized MediaProjection session, both without receipts.

Verified on the API 35 emulator against a Debug server: ASCII and GBK Chinese
text round-tripped through the clipboard into a focused Settings search field,
type / expect_hwnd / window_hwnd returned -32005, and PRINTSCREEN produced a
saved screenshot. check_platform_table.py and verify_tools.py --offline pass.

Co-Authored-By: deepseek-flash
This commit is contained in:
yuanyuanxianganddeepseek-flash committed 2026-11-11 20:59:10 +01:00
1 parent ac43ad92ef
commit 0a0b709e8a
9 files changed
+351 -179

No files matched your search

@@ -4,6 +4,9 @@ import android.app.Notification
import android.app.NotificationChannel
import android.app.NotificationManager
import android.app.Service
import android.content.ClipData
import android.content.ClipboardManager
import android.content.Context
import android.content.Intent
import android.hardware.display.DisplayManager
import android.hardware.display.VirtualDisplay
@@ -23,6 +26,8 @@ import android.util.Log
import android.view.WindowManager
import android.widget.Toast
import java.nio.charset.Charset
class CaptureService : Service() {
companion object {
@@ -59,6 +64,36 @@ class CaptureService : Service() {
UpdateInstaller.onApkReceived(ctx, path)
}
/**
* 远程剪贴板(COMMAND_SCREEN_SET_CLIPBOARD):服务端按 GBK/ANSI 下发字节(Windows
* CF_TEXT 语义),这里解成字符串写系统剪贴板。写入不要求应用在前台(Android 10+
* 只限制「读」剪贴板),所以主控能往后台的客户端塞文本;粘到目标输入框由用户在
* 远端的输入框里用 VK_V(本客户端映射为 ACTION_PASTE)完成。
*/
@JvmStatic
fun onClipboardText(bytes: ByteArray) {
val ctx = instance ?: appContext
if (ctx == null) {
Log.w(TAG, "onClipboardText: no context, ${bytes.size} bytes dropped")
return
}
val cm = ctx.getSystemService(Context.CLIPBOARD_SERVICE) as? ClipboardManager
if (cm == null) {
Log.e(TAG, "onClipboardText: no ClipboardManager")
return
}
// 人类剪贴板同步(ScreenSpyDlg / HideScreenSpyDlg)发的是带 '\0' 结尾的 CF_TEXT,
// MCP 路径不带;统一在这里去掉尾随 NUL,否则粘进输入框会多一个不可见字符
val text = String(bytes, Charset.forName("GBK")).trimEnd('\u0000')
try {
cm.setPrimaryClip(ClipData.newPlainText("YAMA", text))
Log.i(TAG, "onClipboardText: ${bytes.size} bytes, ${text.length} chars")
} catch (e: Throwable) {
// Android 10+ 极少见地会在某些受限场景抛 SecurityException;记日志不崩 JNI 线程
Log.e(TAG, "onClipboardText: setPrimaryClip failed: $e")
}
}
@JvmStatic
fun onNativeExit() {
val svc = instance ?: return
@@ -436,9 +436,48 @@ class ControlService : AccessibilityService() {
0x24 -> performGlobalAction(GLOBAL_ACTION_HOME) // VK_HOME
0x5D -> performGlobalAction(GLOBAL_ACTION_RECENTS) // VK_APPS
0x2C -> performGlobalAction(GLOBAL_ACTION_TAKE_SCREENSHOT) // PrtSc
0x56 -> pasteIntoFocused() // VK_V → 粘贴
}
}
// 远程剪贴板的粘贴动作。Android 没有 Ctrl+V 组合键,而且本客户端不注入文本,
// 所以把 VK_V 映射成输入框的 ACTION_PASTE:由目标应用自己读剪贴板,
// 不受 Android 10+「后台应用不能读剪贴板」的限制。
@Suppress("DEPRECATION")
private fun pasteIntoFocused() {
val root = rootInActiveWindow ?: run {
Log.w(TAG, "paste: no active window")
return
}
// 焦点链:输入焦点 → 无障碍焦点 → 整棵树里第一个可编辑节点。部分应用只置
// FOCUS_ACCESSIBILITY;也不拿 isEditable 当门槛(WebView / 自绘输入框常常不置位
// 但照样响应 ACTION_PASTE),真正的判据是 performAction 的返回值。
val target = root.findFocus(AccessibilityNodeInfo.FOCUS_INPUT)
?: root.findFocus(AccessibilityNodeInfo.FOCUS_ACCESSIBILITY)
?: findFirstEditable(root)
if (target == null) {
Log.w(TAG, "paste: no focused or editable node")
} else {
if (!target.performAction(AccessibilityNodeInfo.ACTION_PASTE))
Log.w(TAG, "paste: ACTION_PASTE rejected by ${target.packageName}")
target.recycle()
}
root.recycle()
}
// 找不到焦点时的兜底:返回整棵树里第一个可见的可编辑节点(调用方负责 recycle)
@Suppress("DEPRECATION")
private fun findFirstEditable(node: AccessibilityNodeInfo): AccessibilityNodeInfo? {
if (node.isEditable && node.isVisibleToUser) return AccessibilityNodeInfo.obtain(node)
for (i in 0 until node.childCount) {
val child = node.getChild(i) ?: continue
val hit = findFirstEditable(child)
child.recycle()
if (hit != null) return hit
}
return null
}
private fun dispatchTap(x: Float, y: Float) {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) return
val path = Path().apply { moveTo(x, y); lineTo(x + 1f, y) }