GitHub Copilot can now interact with desktop apps with computer use
Quick Answer
GitHub Copilot now supports desktop app interaction in public preview, allowing automation of workflows in applications without APIs.
Quick Take
Users maintain control with approval prompts, and it works on macOS and Windows, enhancing productivity in legacy software environments.
Key Points
- Copilot can read app content, click controls, and navigate workflows.
- Users must approve Copilot's actions before it interacts with apps.
- Feature can be disabled by organization-managed settings.
- Works best when users specify desired outcomes and involved applications.
- Available in GitHub Copilot CLI and the app on macOS and Windows.
📖 Reader Mode
~2 min readComputer use is now available in public preview in GitHub Copilot CLI and the GitHub Copilot app on macOS and Windows.
Copilot can interact with desktop applications on your behalf (e.g., reading accessible app content and visual context, clicking controls, entering and editing text, pressing keys, scrolling, dragging, and navigating workflows across applications). This expands the tasks Copilot can help automate, including workflows in legacy and GUI-only software that do not provide an API, command-line interface, or MCP integration.
Copilot uses computer-use tools to navigate an expense-report workflow in Safari.
You remain in control. Copilot asks for approval before controlling an app, and you can review or reset apps that you have chosen to always allow. On macOS, computer use also guides you through the required Accessibility and Screen Recording permissions. Organization-managed settings can disable the feature.
To get started:
- In Copilot CLI, run
/computer on. Use/computer showto check its status and/computer offto disable it. - In the GitHub Copilot app, open Settings, select Computer Use, and turn on Enable Computer Use. You can also use
/computer on.
Computer use works best when you describe the outcome you want, the applications involved, and any important constraints. For example, you can ask Copilot to summarize notifications in a browser, update content in a presentation, or move information through a workflow in a desktop application.
Learn more about computer use in GitHub Copilot CLI and the GitHub Copilot app
— Originally published at github.blog
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from GitHub Copilot Changelog
See more →
GitHub Copilot for JetBrains adds improved OpenTelemetry configuration and model management
The latest GitHub Copilot for JetBrains update enhances workflow control with OpenTelemetry configuration, improved model management, and support for servers in Claude agent flows, facilitating better observability and cost control for enterprise users.
