["# Linux GUI Control 技能","## 概述","利用 xdotool、wmctrl 和 dogtail 控制 Linux 桌面 GUI。","## 用途","- 与非浏览器应用交互","- 模拟鼠标/键盘输入","- 管理窗口","- 检查 X11/GNOME 应用的 UI 层次结构","## 支持的功能","- 在应用中点击/输入","- 调整/移动窗口","- 提取应用的基于文本的 UI 树（A11y）","- 捕获屏幕截图进行视觉分析","## 示例用法","> 注意： 由于原始内容中未提供具体的示例用法，此处保持空白。实际翻译时，请根据原始技能的 SKILL.md 添加具体示例。"]
This skill provides tools and procedures for automating interactions with the Linux desktop environment.
Quick Start
1. Identify Target Window
Use
wmctrl to find the exact name of the window you want to control.
wmctrl -l
2. Inspect UI Hierarchy
For apps supporting accessibility (GNOME apps, Electron apps with
--force-renderer-accessibility), use the inspection script to find button names without taking screenshots.
python3 scripts/inspect_ui.py ""
3. Perform Actions
Use
xdotool via the helper script for common actions.
# Activate window
./scripts/gui_action.sh activate ""# Click coordinates
./scripts/gui_action.sh click 500 500
# Type text
./scripts/gui_action.sh type "Hello World"
# Press a key
./scripts/gui_action.sh key "Return"
Workflows
Operating an App via Text UI
- List windows with
wmctrl -l.
- Activate the target window.
- Run
scripts/inspect_ui.py to get the list of buttons and inputs.
- Use
xdotool key Tab and Return to navigate, or click if coordinates are known.
- If text-based inspection fails, fallback to taking a screenshot and using vision.
Forcing Accessibility in Electron Apps
Many modern apps (VS Code, Discord, Cider, Chrome) need a flag to expose their UI tree:
pkill
nohup --force-renderer-accessibility > /dev/null 2>&1 &
Tool Reference
- wmctrl: Window management (list, activate, move, resize).
- xdotool: Input simulation (click, type, key, mousemove).
- dogtail: UI tree extraction via AT-SPI (Accessibility bus).
- scrot: Lightweight screenshot tool.