openai · OpenAI Platform Docs
Graders | OpenAI API
Teaches how to implement LLM-based grading patterns to evaluate the quality, accuracy, or adherence of model outputs against specific criteria.
Derived skill
Files assembled from official documentation
Viewing SKILL.md
Graders | OpenAI API
Teaches how to implement LLM-based grading patterns to evaluate the quality, accuracy, or adherence of model outputs against specific criteria.
When To Use
Use when you need to automate the evaluation of model outputs using a secondary LLM to score performance or verify compliance with specific instructions.
Reference Files
| File | Contains | Use For |
|---|---|---|
SKILL.md | Entry point: scope, routing table, and workflow. | Start here. |
docs/graders-openai-api-workflow-guide.md | A technical guide detailing the implementation of multigraders, design tips, and technical constraints for using the OpenAI API for grading tasks. | Questions about a technical guide detailing the implementation of multigraders, design tips, and technical constraints for using the... |
examples/graders-openai-api-openai-api-graders.text | A text-based example demonstrating the structure of a grader response including lead, excerpt, and reference answer fields. | Exact payloads, commands, or snippets shown in A text-based example demonstrating the structure of a grader response including lead, excerpt, and reference answer f... |
examples/graders-openai-api-openai-api-graders-json-configuration.text | A JSON configuration object defining multi-type grader rules and stringcheck operations for evaluating API outputs. | Exact payloads, commands, or snippets shown in A JSON configuration object defining multi-type grader rules and stringcheck operations for evaluating API outputs. |
examples/graders-openai-api-openai-api-graders-definition.text | A text representation of the JSON schema structure used for defining grading operations and string comparison rules in the OpenAI API. | Exact payloads, commands, or snippets shown in A text representation of the JSON schema structure used for defining grading operations and string comparison rules i... |
examples/graders-openai-api-openai-api-graders-textsimilarity.text | A text-based schema definition for the textsimilarity grader type including evaluation metrics like bleu, rouge, and cosine similarity. | Exact payloads, commands, or snippets shown in A text-based schema definition for the textsimilarity grader type including evaluation metrics like bleu, rouge, and... |
examples/graders-openai-api-openai-api-grader-scoremodel.text | A text representation of the scoremodel JSON schema used for defining grader configurations in the OpenAI API. | Exact payloads, commands, or snippets shown in A text representation of the scoremodel JSON schema used for defining grader configurations in the OpenAI API. |
examples/graders-openai-api-openai-api-graders-text-format.text | A text representation of the grading structure including roles and content excerpts for the OpenAI API. | Exact payloads, commands, or snippets shown in A text representation of the grading structure including roles and content excerpts for the OpenAI API. |
examples/graders-openai-api-openai-api-graders-scoremodel-python.text | A Python code example demonstrating how to define a dummy scoremodel grader for the OpenAI API. | Exact payloads, commands, or snippets shown in A Python code example demonstrating how to define a dummy scoremodel grader for the OpenAI API. |
examples/graders-openai-api-openai-api-graders-json.text | A text representation of the JSON schema used for defining grader output structures including results and reasoning steps. | Exact payloads, commands, or snippets shown in A text representation of the JSON schema used for defining grader output structures including results and reasoning s... |
examples/graders-openai-api-openai-api-graders-text-format-2.text | A text file demonstrating the structured format for grader outputs including lead, excerpt, description, and conclusion. | Exact payloads, commands, or snippets shown in A text file demonstrating the structured format for grader outputs including lead, excerpt, description, and conclusion. |
examples/graders-openai-api-openai-api-graders-comparison-format.text | A text representation of the comparison format used by the OpenAI API graders to evaluate ranked answers. | Exact payloads, commands, or snippets shown in A text representation of the comparison format used by the OpenAI API graders to evaluate ranked answers. |
examples/graders-openai-api-openai-api-graders-model-grader-comparison-logic.text | A text representation of the model_grader function logic used to compare multiple answers against a reference answer. | Exact payloads, commands, or snippets shown in A text representation of the modelgrader function logic used to compare multiple answers against a reference answer. |
examples/graders-openai-api-openai-api-graders-python-implementation.text | A Python function demonstrating how to implement a grading logic for the OpenAI API graders guide. | Exact payloads, commands, or snippets shown in A Python function demonstrating how to implement a grading logic for the OpenAI API graders guide. |
examples/graders-openai-api-openai-api-graders-python-implementation-2.text | A Python code snippet demonstrating the implementation of a grading function for the OpenAI API graders guide. | Exact payloads, commands, or snippets shown in A Python code snippet demonstrating the implementation of a grading function for the OpenAI API graders guide. |
examples/graders-openai-api-openai-api-graders-format-examples.text | A text file demonstrating various output formats including text, JSON, tools, and audio for the OpenAI API graders guide. | Exact payloads, commands, or snippets shown in A text file demonstrating various output formats including text, JSON, tools, and audio for the OpenAI API graders gu... |
examples/graders-openai-api-openai-api-graders-format.text | A text representation of the grading data structure including reference answers and scoring keys for the OpenAI API graders guide. | Exact payloads, commands, or snippets shown in A text representation of the grading data structure including reference answers and scoring keys for the OpenAI API g... |
examples/graders-openai-api-openai-api-graders-python-grading-function-implementa.text | A Python implementation of a grading function using the RapidFuzz library to compare sample text against items. | Exact payloads, commands, or snippets shown in A Python implementation of a grading function using the RapidFuzz library to compare sample text against items. |
examples/graders-openai-api-openai-api-graders-python-implementation-3.text | A Python code snippet demonstrating how to implement and use the grader module within the OpenAI API framework. | Exact payloads, commands, or snippets shown in A Python code snippet demonstrating how to implement and use the grader module within the OpenAI API framework. |
examples/graders-openai-api-openai-api-graders-python-dependencies.text | A list of required Python library versions including numpy, scipy, and scikit-learn for implementing OpenAI API graders. | Exact payloads, commands, or snippets shown in A list of required Python library versions including numpy, scipy, and scikit-learn for implementing OpenAI API graders. |
examples/graders-openai-api-openai-api-graders-punkt-nltk-data.text | A text file containing NLTK data components including punkt stopwords and wordnet used for the graders guide. | Exact payloads, commands, or snippets shown in A text file containing NLTK data components including punkt stopwords and wordnet used for the graders guide. |
examples/graders-openai-api-openai-api-graders-2.text | A text-based example demonstrating the structure of grader data including scores and metadata for evaluating model outputs. | Exact payloads, commands, or snippets shown in A text-based example demonstrating the structure of grader data including scores and metadata for evaluating model ou... |
examples/graders-openai-api-openai-api-graders-configuration-json.text | A JSON configuration object defining multi-type graders including text similarity and string checks for evaluating model outputs. | Exact payloads, commands, or snippets shown in A JSON configuration object defining multi-type graders including text similarity and string checks for evaluating mo... |
What This Skill Covers
- get the API key from environment apikey = os.environ["OPENAIAPIKEY"] headers = {"Authorization": f"Bearer {apikey}"}
- Main sections:
Technical constraints,Multigraders,Limitations and tips,Design tips.
Workflow
- Open the most relevant file under
docs/for the exact documented workflow and wording. - Open
schemas/files for exact structured contracts. - Open
examples/files for concrete requests, commands, snippets, and manifests. - Do not add behavior or configuration that is not present in the attached source files.
Canonical source: https://developers.openai.com/api/docs/guides/graders