Files
SimpleRemoter/server
yuanyuanxianganddeepseek-flash ebc2d03290 fix(web): keep the IME bar still and the pointer honest in portrait
Two separate faults showed up once the "I" shortcut was in use in portrait: the
phone input bar would not stay put, and the pointer stopped being where it
claimed to be.

#ime-bar was the last flex child of #screen-page, so it sat at the bottom of the
page, and the only thing keeping it above the keyboard was adjustImeBar()
writing padding-bottom on #screen-page -- the bar chased the keyboard height
frame by frame through visualViewport. The keyboard animation is asynchronous
and the events lag behind it, so while it rose the bar was still low with the
keyboard already covering everything from y=544. A later attempt clamped it
against keyboardTopY() and reserved the last measured keyboard height before the
keyboard arrived. That kept it visible, but the bar still moved whenever the
keyboard changed height -- switching input method, the QuickType row, the
AutoFill Contact chip appearing or going away -- and that is what "the input box
keeps getting pushed up" actually was.

The bar is now a fixed overlay at the bottom edge of the picture, placed by the
same formula the shortcut row already used: picture bottom + 4px. In portrait
adjustImeBar() has nothing left to chase -- it only clears the padding that
landscape may have left behind, and recomputes when the keyboard top changes.
That recompute is idempotent and the style is only written when the value
differs, so in the normal case it produces no movement at all. lastKbHeight,
KB_HEIGHT_GUESS, noteKeyboardHeight() and the old keyboard-top tracking are gone
with it.

One clamp survives, as a safety net: if the picture bottom is itself inside the
keyboard block -- a remote machine that is portrait, where the picture is taller
than the space above the keyboard -- the bar falls back to the keyboard top,
because there is nowhere else to put it. That case also has to follow the
keyboard, which is why the recompute above still exists. For a landscape/16:9
remote the clamp never fires and the position is a constant.

Getting that constant right needed a coordinate fix. #ime-bar hangs off
#screen-page while #input-shortcuts lives inside .canvas-container, so the two
have different containing blocks. Reusing the shortcut formula
(canvas.offsetTop + ...) put the bar a whole toolbar height too high -- 279
against a picture that ends at 319 -- which is why it always looked like it was
floating above the shortcut row. The bar's position now comes from the canvas
rect minus the #screen-page rect, so the containing block is whatever the bar
actually resolves against; that subtraction also cancels the translateY
compensation, since both rects are inside the translated element. It lands at
323..382, 4px under the picture.

Landscape keeps the flex behaviour via (orientation: landscape). That is not
just tidiness: an absolutely positioned child does not take padding into
account, because bottom: 0 resolves against the padding box, which includes the
padding. Left as an overlay, the bar would have stayed at the very bottom of the
screen and been covered by the keyboard instead of being pushed up by
padding-bottom.

Opening the keyboard must not move the picture either, and two things were
moving it:

- #screen-page had no overflow guard. #term-page already carries one for exactly
  this reason ("stop iOS from quietly adding a scroll to body and lifting the
  whole page"). Without it iOS hands the page a scroll and then scrolls it once
  the keyboard comes up.
- Even with the page locked, iOS moves the *visual* viewport up when the keyboard
  appears (visualViewport.offsetTop > 0). #screen-page is position: fixed and is
  therefore pinned to the layout viewport, so the entire page -- picture
  included -- was pushed off the top of the screen.

So #screen-page gets overflow: hidden + overscroll-behavior: contain, and
syncVisualOffset() translates it by exactly visualViewport.offsetTop to cancel
the shift. It is driven by the IME rAF loop and the viewport scroll event, and
clears the transform once the offset returns to 0. Nothing downstream has to
compensate for it: the picture, the pointer overlay and the bar are all
descendants of #screen-page, so they move with it.

The pointer overlay was the one thing that did not accept that. .cursor-overlay
was position: fixed, placed by taking the canvas's getBoundingClientRect() and
then subtracting whatever translateY the page currently carried -- two
independent sources of truth that have to agree exactly, the browser's idea of
where the canvas is and our own idea of how far we moved it. They do agree in
desktop Chrome, which is why the harness never saw the fault, but on the phone
they did not: once the IME was open the overlay was drawn off by the
compensation amount while taps still landed correctly, and closing the input box
-- offsetTop back to 0, transform gone -- put it back. The overlay is now
position: absolute inside .canvas-container, the same containing block the canvas
resolves against, and style.left/top hold canvas-local CSS pixels (bx/kx, by/ky).
No viewport rect, no compensation term: whatever an ancestor does to the picture,
it does to the pointer. pageShiftY() is gone with it.

The overlay also has to follow a re-fit, not just a move. resizeScreenCanvas()
keeps the last remote coordinate the overlay was placed at and replays it once
the bitmap has been resized, so the keyboard shrinking the layout viewport, a
rotation or a fullscreen toggle no longer leaves the pointer where the previous
geometry put it.

The touch mapping and the drawing path have to stay exact inverses --
positionCursorOverlay() turns remote coordinates into a position on the canvas,
screenToCanvas() turns a finger position back into remote coordinates -- otherwise
the cursor is drawn where nobody tapped. It was only after the keyboard had been
up for a while that this broke: switching the input method changes the keyboard
height, iOS pans the visual viewport to keep the focused textarea visible, and
from then on offsetTop is non-zero.

iOS also stacks a row of its own directly above the keyboard: the QuickType
predictions and, when Safari decides the field looks like it can take contact
info, an "AutoFill Contact" chip. The textarea gets the full set of suppression
attributes (autocomplete, autocorrect and autocapitalize off, spellcheck off,
inputmode text). That lowers the odds of the chip showing up at all, but iOS
does not promise to honour it -- the only guaranteed off switch is Settings >
Safari > Autofill > Use Contact Info, which is a per-device setting.

With the bar no longer able to hide under the keyboard, the shortcut row below
the picture is only in the way: that row exists to be tapped *before* the IME
opens, and while the IME is open the bar takes its place. Opening the IME now
hides it, so the bar sits 4px under the picture instead of queuing up below the
row. The class lives on #screen-page rather than being stripped off .visible,
because updateUIForOrientation() recomputes .visible from keyboardOpen and would
put the row straight back. Closing the IME -- the bar's own collapse button does
that -- or disconnecting restores it. The key buttons it holds stay reachable by
reopening the mobile keyboard, which is what showed the row in the first place.

The bar also has to exist only while the keyboard does. Tapping it is what wakes
the keyboard in the first place, and iOS then scrolls the page to keep the
focused field visible -- which is the "tap the input box and it shoves itself up"
report. updateImePresence() runs in the IME rAF loop and puts .ime-waiting on the
bar the moment the keyboard is not up -- opacity 0, so it is not there to touch
-- and closeIme() 450ms later once the keyboard has really gone. Two deliberate
choices in that CSS: it is opacity, not display or visibility, because the
textarea has to stay in the document and focusable or the keyboard never comes
up at all; and there is no pointer-events: none, because that would let touches
on that strip fall through to the canvas and become stray clicks on the remote
machine. The wait for the keyboard to appear is capped at 3000ms, after which the
IME is abandoned rather than left invisible forever. It is generous on purpose:
iOS sometimes does not update visualViewport until something scrolls, and a short
timeout would close a perfectly good IME on a slow device.

The RDP reset button redraws the whole frame, but it left the IME open. The bar
stayed on screen, so touching its textarea brought the keyboard back up -- and the
page got shoved again, with the pointer going with it. Reset now closes the IME
first. Since "close the IME" was being spelled out in three places (the I button,
the keyboard toggle, disconnect) and each copy had drifted a little, it is now one
closeIme() that toggleIme(), toggleKeyboard(), sendRdpReset() and disconnect()
all call. The cleanup half runs unconditionally -- imeMode and the DOM can fall out
of sync and a half-closed state is exactly what leaves a bar that pops the
keyboard -- while updateKeyboardButtons(false) is guarded by whether the IME was
actually open, so resetting with only the mobile keyboard up does not close it.
disconnect() additionally keeps its own updateKeyboardButtons(false), because
there the keyboard should close regardless, and it no longer needs to reset
imeMode by hand -- which also stops imeWatcher's rAF loop from spinning on after
a disconnect.

Deselecting the mouse (the control toggle) now drops the keyboard and the IME as
well, for the same reason: leaving the bar on screen means the next touch on its
textarea brings the keyboard back up and shoves the page. Closing only the IME
would not be enough here -- the mobile keyboard is a separate focus -- so
closeKeyboardAndIme() blurs both and is what the control toggle and disconnect()
now share.

Verified with visualViewport.height mocked so the keyboard arrives after the bar
has been placed, on 390x844 and 375x667, with a 16:9 and a portrait remote, at
keyboard heights 0/216/260/300/304/344. For the 16:9 remote the bar is 323..382
at every single step -- zero jump across the whole sweep, where the portrait
remote used to jump 300px. For a portrait remote it tracks the keyboard top
(485..544 at 300px), the only position available to it. No case leaves the bar
under the keyboard, and every measured position equals min(picture bottom + 4,
keyboard top - bar height) to 0.0px.

The bar is present exactly when the keyboard is, checked at every step: on tap
(before the keyboard rises) opacity 0; keyboard up, opacity 1; keyboard down 60ms,
opacity 0 again; 580ms, imeMode false and display none. Reopening with the keyboard
never arriving keeps it at opacity 0 and gives up after 3s. No page errors.
Picture placement across all five layouts is byte-identical to before the change.
Cursor accuracy is dx=dy=0 over portrait/landscape x scale 1/2/3 x offsetTop
0/150/200/300, and the round-trip test -- positionCursorOverlay() asked where to
draw, that screen position fed back through screenToCanvas() -- returns the
original remote point for all nine cases; before the inverse fix six of them were
off, by exactly offsetTop scaled into remote pixels (739 at offsetTop 150 for the
(1800,1000) point).

Tapping RDP reset with the IME open (keyboard mocked at 544, visualViewport
offsetTop at 200) leaves imeMode false, the bar display:none, the transform
cleared, visualOffsetY 0 and the watcher stopped, with the picture still at
y=100..319.4 instead of being shoved up; disconnect() from the same state clears
it identically. Deselecting the mouse from that same state (control on, mobile
keyboard focused, IME open, keyboard mocked at 544 and offsetTop at 180) leaves
controlEnabled false, imeMode false, neither element focused, the bar
display:none, the transform cleared and the watcher stopped, with the picture
still at y=100..319.4. Opening the IME moves the bar from the shortcut row's slot
to 323..382 and closing it puts the row back at 321..369, through both toggleIme()
and a direct class toggle. In landscape the flex path still works: with a 200px
padding the bar goes from 331..390 to 131..190, i.e. just above the keyboard.

The pointer was a second, unrelated bug, and a worse one. screenToCanvas() added
visualViewport.offsetTop on top of a clientX/clientY that is
already expressed in the same coordinate space as getBoundingClientRect(), i.e.
after the translateY compensation. Both quantities include the pan, so subtracting
them already cancels it; adding offsetTop counted it twice. That pushed every
computed remote y down by offsetTop scaled into remote pixels -- 492 remote pixels
at offsetTop 100 on a 16:9 remote -- so everything below the middle of the picture
overshot streamH and was clamped to streamH-1. The symptom is exactly what it
looks like: the pointer can only be moved in the upper part of the picture, a
fixed horizontal band cannot be crossed downwards, and clicking inside that band
lands on the remote taskbar. At offsetTop 300 the whole picture mapped past the
bottom, so every touch landed on the taskbar. The pinch origin had the same
mistake, and so did the cursor recomputation at the end of a pinch, which fed the
overlay's style.top -- a pre-compensation coordinate -- back into screenToCanvas.

Neither direction may add the compensation, and neither does: clientX/clientY and
getBoundingClientRect() are already in the same space, and the overlay no longer
works in screen coordinates at all. The round-trip test had been written against
the wrong premise -- it fed style.top straight into screenToCanvas() -- so it
kept passing while the absolute mapping was wrong. It now converts the overlay's
style values back through the canvas rect on the way in.

Verified against the picture's own rect rather than against the other function:
for points at 10/30/50/70/90/99% of the picture height, screenToCanvas() must
return the remote pixel that sits under that point. At offsetTop 0/100/300 all
eighteen cases are exact (error 0,0); before the fix, offsetTop 100 was off by
+492 on every point and the lower half clamped to 1079, and offsetTop 300 clamped
the entire picture to 1079. The overlay still renders where it should -- for a
pointer driven to the remote bottom it lands 4px above the picture's bottom edge,
the constant hotspot offset, at offsetTop 0 and 300 alike -- and the round-trip
test still returns the original remote point for all nine cases once the
transform is accounted for. With the overlay anchored to the canvas it holds that
alignment through the whole IME lifecycle: dx=dy=0 at offsetTop 0, 200 and 300, on
both a 16:9 remote in portrait and after rotating to landscape and back, and the
pointer lands back on the picture after the IME closes. The rotation case also
covers the re-fit path: the container changes size, the canvas is resized, and
the overlay is replayed at the new geometry instead of staying where the old one
put it.

Co-Authored-By: deepseek-flash
2026-10-31 13:37:27 +01:00
..