Image understanding

Send an image as conversation input and receive a text description.

This guide covers image input with text output, not image generation. Select a model supporting both image input and Chat Completions. A model ID in the list alone does not establish vision support.

Two image sources

• HTTPS image URL: use an image address directly accessible to the gateway or upstream. Browser access does not prove server access; login cookies, temporary authorization, or redirects may cause failures.
• Base64 data URL: read a local image and encode it with its actual MIME type, such as data:image/png;base64,.... Public image hosting is unnecessary, but the request body grows.

Do not use a local path or file:// as a remote image URL. Remote access, formats, dimensions, and image counts depend on the deployment and model.

Runnable example: send a local image

Follow the Python SDK guide, then set one additional environment variable:

export API_IMAGE_PATH='/absolute/path/to/image.png'

Replace the path with an image you are authorized to submit. Save as vision.py and run python vision.py. The command sends the image to the configured API service, which may forward it upstream. This example accepts PNG/JPEG up to 5 MiB; that is an example-specific limit, not a gateway or model allowance.

import base64
import os
from pathlib import Path
from openai import OpenAI, APIConnectionError, APIStatusError

# This example's local limit is not the gateway or model's documented limit.
image_limit = 5 * 1024 * 1024
with Path(os.environ["API_IMAGE_PATH"]).open("rb") as image_file:
    image = image_file.read(image_limit + 1)
if not image or len(image) > image_limit:
    raise SystemExit("Choose a nonempty image up to 5 MiB for this example.")
if image.startswith(b"\x89PNG\r\n\x1a\n"):
    mime_type = "image/png"
elif image.startswith(b"\xff\xd8\xff"):
    mime_type = "image/jpeg"
else:
    raise SystemExit("This example accepts PNG or JPEG images only.")
image_url = f"data:{mime_type};base64,{base64.b64encode(image).decode('ascii')}"

with OpenAI(
    base_url=os.environ["API_BASE_URL"].rstrip("/"),
    api_key=os.environ["API_KEY"],
    timeout=60.0,
    max_retries=0,
) as client:
    try:
        result = client.chat.completions.create(
            model=os.environ["API_MODEL"],
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this image in one sentence."},
                    {"type": "image_url", "image_url": {"url": image_url}},
                ],
            }],
        )
        choice = result.choices[0] if result.choices else None
        if (choice is None or choice.finish_reason != "stop"
                or choice.message.refusal or choice.message.tool_calls
                or not choice.message.content):
            raise SystemExit("Image request did not produce a complete text reply.")
        print(choice.message.content)
    except APIStatusError as error:
        raise SystemExit(f"Request failed (HTTP {error.status_code}). Check the request logs.")
    except APIConnectionError:
        raise SystemExit("Connection failed. Check the API address and network.")

The example identifies MIME from the file header; it does not validate full image decodability. Services can additionally limit pixels, encoding, and total request size. Base64 payloads are larger than the original file. Avoid logging complete data URLs.

Protocol field differences

Item	Chat Completions	Responses
Text block	type: text, text in text	type: input_text, text in text
Image block	type: image_url, image_url is an object containing url	type: input_image, image_url is a string

Switching to Responses also requires changing the input structure. Optional fields such as detail require model support; validate a request without extra options first.

Python: vision with Responses

Save as responses-vision.py, use the Python environment and API_IMAGE_PATH from above, and run python responses-vision.py. This submits the image to your configured service; choose a model and channel supporting Responses image input. The example keeps the same local 5 MiB file limit, uses input_text and input_image, and does not require a separate Files API upload.

import base64
import os
from pathlib import Path
from openai import OpenAI, APIConnectionError, APIStatusError

# This example's local limit is not the gateway or model's documented limit.
image_limit = 5 * 1024 * 1024
with Path(os.environ["API_IMAGE_PATH"]).open("rb") as image_file:
    image = image_file.read(image_limit + 1)
if not image or len(image) > image_limit:
    raise SystemExit("Choose a nonempty image up to 5 MiB for this example.")
if image.startswith(b"\x89PNG\r\n\x1a\n"):
    mime_type = "image/png"
elif image.startswith(b"\xff\xd8\xff"):
    mime_type = "image/jpeg"
else:
    raise SystemExit("This example accepts PNG or JPEG images only.")
image_url = f"data:{mime_type};base64,{base64.b64encode(image).decode('ascii')}"

with OpenAI(
    base_url=os.environ["API_BASE_URL"].rstrip("/"),
    api_key=os.environ["API_KEY"],
    timeout=60.0,
    max_retries=0,
) as client:
    try:
        result = client.responses.create(
            model=os.environ["API_MODEL"],
            input=[{
                "role": "user",
                "content": [
                    {"type": "input_text", "text": "Describe this image in one sentence."},
                    {"type": "input_image", "image_url": image_url},
                ],
            }],
        )
        if result.status != "completed" or not result.output_text.strip():
            raise SystemExit("Image request did not produce a complete text reply.")
        for item in result.output:
            if item.type == "reasoning":
                continue
            if (item.type != "message" or item.status != "completed"
                    or any(part.type != "output_text" for part in item.content)):
                raise SystemExit("Image request did not produce a complete text reply.")
        print(result.output_text)
    except APIStatusError as error:
        raise SystemExit(f"Request failed (HTTP {error.status_code}). Check the request logs.")
    except APIConnectionError:
        raise SystemExit("Connection failed. Check the API address and network.")

As in the Chat example, a file signature check is not full image decoding. The response must be complete with nonempty text; refusals, incomplete messages, and tool items are not published as image descriptions.

Troubleshooting

Symptom	Check
Rejected content type or parameter	Model vision support, endpoint protocol, and channel conversion support
Unreadable URL	Server reachability, authorization, URL expiry, and image content type
413 or oversized request	Proxy, gateway, and upstream limits; resize or compress appropriately in the application
Empty or truncated text	Completion reason, refusal information, and model response format
Inaccurate interpretation	Image clarity, text size, and question specificity; verify manually when needed

HTTP integration · Errors and retries

Reference: OpenAI image inputs.

For generating or editing images, see image generation.

简体中文