openai · OpenAI Platform Docs

Overview of OpenAI Crawlers

Explains how to manage OpenAI web crawlers using robots.txt tags to control how OAI-SearchBot, OAI-AdsBot, and GPTBot interact with website content for search, advertising, and model training.

Import to Prompt Buddy

Derived skill

Files assembled from official documentation

Viewing SKILL.md

Overview of OpenAI Crawlers

Explains how to manage OpenAI web crawlers using robots.txt tags to control how OAI-SearchBot, OAI-AdsBot, and GPTBot interact with website content for search, advertising, and model training.

When To Use

Use when you need to configure robots.txt settings to permit or restrict specific OpenAI user agents from crawling your site for search indexing, ad validation, or AI model training.

Reference Files

File	Contains	Use For
`SKILL.md`	Entry point: scope, routing table, and workflow.	Start here.
`docs/overview-of-openai-crawlers-workflow-guide.md`	An overview of how OpenAI uses web crawlers like OAI-SearchBot and GPTBot and how to manage them via robots.txt.	Questions about an overview of how OpenAI uses web crawlers like OAI-SearchBot and GPTBot and how to manage them via robots.txt.

What This Skill Covers

OpenAI uses web crawlers (“robots”) and user agents to perform actions for its products, either automatically or triggered by user request. OpenAI uses OAI-S...

Workflow

Open the most relevant file under docs/ for the exact documented workflow and wording.
Open schemas/ files for exact structured contracts.
Open examples/ files for concrete requests, commands, snippets, and manifests.
Do not add behavior or configuration that is not present in the attached source files.

Canonical source: https://developers.openai.com/api/docs/bots.md