openai · OpenAI Platform Docs
Computer use | OpenAI API
Teaches how to implement and use the computer use capability within the OpenAI API to allow models to interact with a computer interface.
Derived skill
Files assembled from official documentation
Viewing SKILL.md
Computer use | OpenAI API
Teaches how to implement and use the computer use capability within the OpenAI API to allow models to interact with a computer interface.
When To Use
Use when you need to implement a workflow where an AI agent interacts with a computer interface via tool calling to perform tasks like clicking, typing, or navigating a desktop environment.
Reference Files
| File | Contains | Use For |
|---|---|---|
SKILL.md | Entry point: scope, routing table, and workflow. | Start here. |
docs/computer-use-openai-api-workflow-guide.md | A guide detailing the implementation of computer use capabilities via the OpenAI API, including screenshot capture and tool execution loops. | Questions about a guide detailing the implementation of computer use capabilities via the OpenAI API, including screenshot capture an... |
examples/computer-use-openai-api-openai-api-computer-use-playwright-chromium-setu.text | A JavaScript code snippet demonstrating how to launch a Chromium browser instance with specific arguments for OpenAI computer use capabilities. | Exact payloads, commands, or snippets shown in A JavaScript code snippet demonstrating how to launch a Chromium browser instance with specific arguments for OpenAI... |
examples/computer-use-openai-api-playwright-python.text | A Python script using Playwright to demonstrate the computer use capability via the OpenAI API. | Exact payloads, commands, or snippets shown in A Python script using Playwright to demonstrate the computer use capability via the OpenAI API. |
examples/computer-use-openai-api-ubuntu-dockerfile.text | A Dockerfile configuration for setting up an Ubuntu 22.04 environment with XFCE4 and necessary dependencies for OpenAI computer use capabilities. | Exact payloads, commands, or snippets shown in A Dockerfile configuration for setting up an Ubuntu 22.04 environment with XFCE4 and necessary dependencies for OpenA... |
examples/computer-use-openai-api-docker-build-command.text | A text snippet containing the docker build command used to create the computer use agent image. | Exact payloads, commands, or snippets shown in A text snippet containing the docker build command used to create the computer use agent image. |
examples/computer-use-openai-api-docker-run-command.text | A docker run command used to launch a container image configured for computer use with a VNC display. | Exact payloads, commands, or snippets shown in A docker run command used to launch a container image configured for computer use with a VNC display. |
examples/computer-use-openai-api-python-vm-execution.text | A Python implementation demonstrating how to use the OpenAI API for computer use by executing commands within a virtual machine environment. | Exact payloads, commands, or snippets shown in A Python implementation demonstrating how to use the OpenAI API for computer use by executing commands within a virtu... |
examples/computer-use-openai-api-docker-exec-javascript.text | A JavaScript code snippet demonstrating how to execute commands within a Docker container to facilitate computer use via the OpenAI API. | Exact payloads, commands, or snippets shown in A JavaScript code snippet demonstrating how to execute commands within a Docker container to facilitate computer use... |
examples/computer-use-openai-api-openai-api-computer-use-nodejs.text | A Node.js code example demonstrating how to use the computer tool within an OpenAI API request to interact with a user interface. | Exact payloads, commands, or snippets shown in A Node.js code example demonstrating how to use the computer tool within an OpenAI API request to interact with a use... |
examples/computer-use-openai-api-python-client.text | A Python code example demonstrating how to use the computer tool within an OpenAI API request to perform UI interactions. | Exact payloads, commands, or snippets shown in A Python code example demonstrating how to use the computer tool within an OpenAI API request to perform UI interacti... |
examples/computer-use-openai-api-openai-api-computer-use-tool-call.text | A text representation of a computer use tool call output containing screenshot action status and call ID. | Exact payloads, commands, or snippets shown in A text representation of a computer use tool call output containing screenshot action status and call ID. |
examples/computer-use-openai-api-openai-computer-use-playwright-key-normalization.text | A JavaScript function that maps model-emitted keyboard action strings to the specific key names required by the Playwright automation library. | Exact payloads, commands, or snippets shown in A JavaScript function that maps model-emitted keyboard action strings to the specific key names required by the Playw... |
examples/computer-use-openai-api-openai-computer-use-playwright-key-normalization-2.text | A Python function that maps model-emitted keyboard key names to the specific string formats required by the Playwright automation library. | Exact payloads, commands, or snippets shown in A Python function that maps model-emitted keyboard key names to the specific string formats required by the Playwrigh... |
examples/computer-use-openai-api-openai-computer-use-xdotool-key-normalization.text | A JavaScript function that maps model-emitted key names to the specific string format required by the xdotool utility. | Exact payloads, commands, or snippets shown in A JavaScript function that maps model-emitted key names to the specific string format required by the xdotool utility. |
examples/computer-use-openai-api-openai-computer-use-xdotool-key-normalization-2.text | A Python function that maps model-emitted key names to the specific key names expected by the xdotool utility. | Exact payloads, commands, or snippets shown in A Python function that maps model-emitted key names to the specific key names expected by the xdotool utility. |
examples/computer-use-openai-api-action-sequence.text | A text representation of a computer use tool call containing a sequence of click and type actions. | Exact payloads, commands, or snippets shown in A text representation of a computer use tool call containing a sequence of click and type actions. |
examples/computer-use-openai-api-openai-api-computer-use-javascript-action-handle.text | A JavaScript function implementation for handling various computer use tool actions such as mouse movements and keyboard inputs. | Exact payloads, commands, or snippets shown in A JavaScript function implementation for handling various computer use tool actions such as mouse movements and keybo... |
examples/computer-use-openai-api-openai-api-computer-use-python-action-handler.text | A Python function implementation for handling computer use tool actions such as mouse clicks and keyboard inputs. | Exact payloads, commands, or snippets shown in A Python function implementation for handling computer use tool actions such as mouse clicks and keyboard inputs. |
examples/computer-use-openai-api-openai-api-computer-use-tool-implementation.text | A text-based implementation example demonstrating how to use the computer use tool within the OpenAI API. | Exact payloads, commands, or snippets shown in A text-based implementation example demonstrating how to use the computer use tool within the OpenAI API. |
examples/computer-use-openai-api-openai-computer-use-python-action-handler.text | A Python function implementation for handling computer use actions within a virtual machine environment. | Exact payloads, commands, or snippets shown in A Python function implementation for handling computer use actions within a virtual machine environment. |
examples/computer-use-openai-api-action.text | A text representation of a computer use tool call output containing click actions and coordinates. | Exact payloads, commands, or snippets shown in A text representation of a computer use tool call output containing click actions and coordinates. |
examples/computer-use-openai-api-openai-api-computer-use-javascript-implementatio.text | A JavaScript implementation demonstrating how to use modifiers with computer use tool functions. | Exact payloads, commands, or snippets shown in A JavaScript implementation demonstrating how to use modifiers with computer use tool functions. |
examples/computer-use-openai-api-openai-api-computer-use-python-implementation.text | A Python code implementation demonstrating how to use the OpenAI API for computer use capabilities. | Exact payloads, commands, or snippets shown in A Python code implementation demonstrating how to use the OpenAI API for computer use capabilities. |
examples/computer-use-openai-api-openai-api-computer-use-guide.text | A text-based guide explaining the implementation and usage of the computer use tool within the OpenAI API. | Exact payloads, commands, or snippets shown in A text-based guide explaining the implementation and usage of the computer use tool within the OpenAI API. |
examples/computer-use-openai-api-openai-api-computer-use-guide-2.text | A text-based walkthrough and conceptual example of implementing computer use capabilities via the OpenAI API. | Exact payloads, commands, or snippets shown in A text-based walkthrough and conceptual example of implementing computer use capabilities via the OpenAI API. |
examples/computer-use-openai-api-nodejs-screenshot-capture.text | A Node.js code snippet demonstrating how to capture a screenshot using a page object for the OpenAI computer use API. | Exact payloads, commands, or snippets shown in A Node.js code snippet demonstrating how to capture a screenshot using a page object for the OpenAI computer use API. |
examples/computer-use-openai-api-python-screenshot-capture.text | A Python code snippet demonstrating how to capture a screenshot of a web page for use with the OpenAI computer use tool. | Exact payloads, commands, or snippets shown in A Python code snippet demonstrating how to capture a screenshot of a web page for use with the OpenAI computer use tool. |
examples/computer-use-openai-api-javascript-screenshot-capture.text | A JavaScript function demonstrating how to capture a screenshot from a virtual machine using Docker execution for the OpenAI computer use API. | Exact payloads, commands, or snippets shown in A JavaScript function demonstrating how to capture a screenshot from a virtual machine using Docker execution for the... |
examples/computer-use-openai-api-python-screenshot-capture-2.text | A Python function demonstrating how to capture a screenshot from a Docker container for the OpenAI computer use tool. | Exact payloads, commands, or snippets shown in A Python function demonstrating how to capture a screenshot from a Docker container for the OpenAI computer use tool. |
examples/computer-use-openai-api-nodejs-screenshot-response.text | A Node.js code example demonstrating how to send a computer screenshot as a tool output using the OpenAI API. | Exact payloads, commands, or snippets shown in A Node.js code example demonstrating how to send a computer screenshot as a tool output using the OpenAI API. |
examples/computer-use-openai-api-openai-api-computer-use-python-implementation-2.text | A Python code example demonstrating how to use the OpenAI API to send computer call outputs and screenshots using the computer tool. | Exact payloads, commands, or snippets shown in A Python code example demonstrating how to use the OpenAI API to send computer call outputs and screenshots using the... |
examples/computer-use-openai-api-openai-api-computer-use-nodejs-loop.text | A Node.js implementation of a continuous loop that processes computer use tool calls from the OpenAI API response. | Exact payloads, commands, or snippets shown in A Node.js implementation of a continuous loop that processes computer use tool calls from the OpenAI API response. |
examples/computer-use-openai-api-openai-api-computer-use-python-implementation-lo.text | A Python code implementation demonstrating a loop for handling computer use tool calls via the OpenAI API. | Exact payloads, commands, or snippets shown in A Python code implementation demonstrating a loop for handling computer use tool calls via the OpenAI API. |
examples/computer-use-openai-api-openai-api-computer-use-guide-3.text | A text-based guide outlining the implementation and usage of the computer use capability within the OpenAI API. | Exact payloads, commands, or snippets shown in A text-based guide outlining the implementation and usage of the computer use capability within the OpenAI API. |
examples/computer-use-openai-api-openai-api-computer-use-guide-4.text | A text-based guide outlining the implementation and capabilities of the computer use tool within the OpenAI API. | Exact payloads, commands, or snippets shown in A text-based guide outlining the implementation and capabilities of the computer use tool within the OpenAI API. |
examples/computer-use-openai-api-definitions-guide.text | A text-based guide defining user versus non-user content safety considerations for the OpenAI Computer Use API. | Exact payloads, commands, or snippets shown in A text-based guide defining user versus non-user content safety considerations for the OpenAI Computer Use API. |
examples/computer-use-openai-api-confirmation-hygiene-guidelines.text | Best practices and guidelines for managing user confirmations and sensitive data transmission when using the computer use tool. | Exact payloads, commands, or snippets shown in Best practices and guidelines for managing user confirmations and sensitive data transmission when using the computer... |
examples/computer-use-openai-api-sensitive-data-transmission-guidelines.text | A text document outlining sensitive data types and transmission security considerations for the OpenAI Computer Use API. | Exact payloads, commands, or snippets shown in A text document outlining sensitive data types and transmission security considerations for the OpenAI Computer Use API. |
examples/computer-use-openai-api-prompt-injection-guidelines.text | Textual guidelines and descriptions regarding prompt injection risks and mitigation strategies for the OpenAI computer use capability. | Exact payloads, commands, or snippets shown in Textual guidelines and descriptions regarding prompt injection risks and mitigation strategies for the OpenAI compute... |
examples/computer-use-openai-api-nodejs-client-response-create.text | A Node.js code example demonstrating how to use the OpenAI client to create a response with the computer-use-preview tool. | Exact payloads, commands, or snippets shown in A Node.js code example demonstrating how to use the OpenAI client to create a response with the computer-use-preview... |
examples/computer-use-openai-api-python-request.text | A Python code snippet demonstrating how to initialize the OpenAI client and call the computer-use-preview model with specific tool configurations. | Exact payloads, commands, or snippets shown in A Python code snippet demonstrating how to initialize the OpenAI client and call the computer-use-preview model with... |
What This Skill Covers
- Reuse normalizekey from the helper above. Reuse normalizedragpath from the helper above.
- Main sections:
4. Capture and return the updated screenshot,5. Repeat until the tool stops calling,Possible Computer use actions,Option 2: Use a custom tool or harness,Option 3: Use a code-execution harness.
Workflow
- Open the most relevant file under
docs/for the exact documented workflow and wording. - Open
schemas/files for exact structured contracts. - Open
examples/files for concrete requests, commands, snippets, and manifests. - Do not add behavior or configuration that is not present in the attached source files.
Canonical source: https://developers.openai.com/api/docs/guides/tools-computer-use