Windows UI Inject

Windows UI inject (browser + RDP client)

Former name: browser-read-bypass. Same skill.

When computer-use can see a window but cannot click or type, inject input with user32 / System.Windows.Forms from PowerShell (run via the shell tool — do not type into a PowerShell window through the UI).

Applies to:

• Browsers (Edge, Chrome, Firefox) under tier read
• mstsc / Remote Desktop Connection (security warning, credentials, session chrome)
• Other non-elevated desktop apps the agent must drive from screenshots

For building a GNOME Wayland / GDM-style RDP server (headless shell + GRD), use gnome-wayland-rdp. Full loop: agent-workstation-drive. This skill is client-side UI automation only. Canvas/WebGL pages when computer-use can click: read-canvas-browser.

Portable script

scripts/Invoke-MstscConnect.ps1 — parameterized host/port/user/password, cmdkey seed, Connect (not Learn more), screenshots. Run with Windows PowerShell 5.1.

Core sequence

1. Screenshot — identify the focused window and targets
2. Dismiss stray menus (ESC) if needed
3. Click (or key-activate) the control
4. Type if needed
5. Screenshot again to verify

Never assume fixed pixel positions across sessions — re-measure from the current shot and window rect. Prefer window-relative fractions after GetWindowRect.

Reusable PowerShell

Click at coordinates

Add-Type -TypeDefinition @'
using System.Runtime.InteropServices;
public class Mouse {
    [DllImport("user32.dll")] public static extern bool SetCursorPos(int x, int y);
    [DllImport("user32.dll")] public static extern void mouse_event(int f, int a, int b, int c, int d);
    public static void Click(int x, int y) {
        SetCursorPos(x, y);
        System.Threading.Thread.Sleep(150);
        mouse_event(2, 0, 0, 0, 0);
        mouse_event(4, 0, 0, 0, 0);
        System.Threading.Thread.Sleep(150);
    }
}
'@
[Mouse]::Click($x, $y)

Type / keys

Add-Type -AssemblyName System.Windows.Forms
[System.Windows.Forms.SendKeys]::SendWait("text")
[System.Windows.Forms.SendKeys]::SendWait("{ENTER}")
[System.Windows.Forms.SendKeys]::SendWait("^a")      # Ctrl+A
[System.Windows.Forms.SendKeys]::SendWait("+{TAB}")  # Shift+Tab

Escape in SendKeys: + ^ % ~ ( ) [ ] { } → wrap in {} (e.g. {+}).

Find a visible window by title (optionally by PID)

# EnumWindows + GetWindowText + optional GetWindowThreadProcessId
# Prefer scoping to mstsc PID so Edge tabs titled "Remote Desktop" are not hit

Use Windows PowerShell 5.1 ($env:SystemRoot\System32\WindowsPowerShell\v1.0\powershell.exe) when Add-Type + System.Drawing fails under PowerShell 7 (missing assembly refs).

Full-screen screenshot

Add-Type -AssemblyName System.Windows.Forms,System.Drawing
# CopyFromScreen PrimaryScreen.Bounds → PNG path

mstsc / RDP client automation

Rules that prevent false negatives

Security warning dialog layout (relative)

After GetWindowRect → width W, height H, origin (L, T):

• Resource checkboxes (left column): roughly L + 0.18*W, mid-dialog height — if a "Use the following credentials" block is present, list is higher (0.55–0.62*H); without it, lower (0.70*H).
• Connect: left of Cancel, ~`L + 0.72*W, ~T + 0.92*H`
• Cancel: default focus (often blue outline), right of Connect
• Learn more: hyperlink in the upper warning body — never aim clicks at mid-body right

Credentials

• Windows Security ("Enter your credentials") is often a different process from mstsc.
• CredUI / password fields frequently ignore SendKeys and Unicode SendInput from automation (empty field, OK no-ops).
• Lab automation pattern:

cmdkey /generic:TERMSRV/<host> /user:<user> /pass:<pass>
# .rdp: full address, username, authentication level:i:0,
#       prompt for credentials:i:0, enablecredsspsupport as needed

Do not bake host, port, or password into skills — take them from the project .rdp, env, or user. Use placeholders in docs: <host>, <port>, <user>.

Session established signals

• mstsc MainWindowTitle becomes something like <file> - <host>[:port] - Remote Desktop Connection
• Pure black client area can mean server-side graphics stall (see gnome-wayland-rdp), not a failed Connect click
• Always check server logs before re-clicking Connect in a loop

Browser navigation (summary)

1. ESC to clear menus
2. Click address bar (measure; ~top chrome center is only a hint)
3. ^a, type URL, {ENTER}
4. Wait for load, screenshot

Timing

• 150–400 ms after focus/click before typing
• 1–2 s after Connect before assuming credentials or session chrome
• If a click misses, re-shot and recompute from rect — lag is more common than wrong API

Limitations

• Windows only
• UAC / elevated windows: UIPI blocks injection from lower integrity
• SendKeys goes to the foreground window — SetForegroundWindow first
• Does not bypass network, TLS, or auth policy — only local UI input
• PowerShell 7 may need different Add-Type assembly refs than 5.1