Sending Images to GPT-4o, Claude, and Gemini: The Base64 Payload Each One Wants

You want to send a screenshot to a vision model. All three of the big ones — OpenAI's GPT-4o, Anthropic's Claude, Google's Gemini — accept images the same fundamental way: Base64-encode the bytes and put them in the JSON request. No file uploads, no multipart, just text in a payload. And yet the single most common error people hit is some flavor of invalid image / could not process image . The reason is almost never the image. It's that each provider wants the Base64 wrapped in a differently shaped object , and the traps are subtle — especially the data: URL prefix, which one provider requires and the other two reject. Here's the exact payload each one wants, side by side. OpenAI (GPT-4o) GPT-4o uses a content array of parts. The image is an image_url part, and — this is the trap — the url field takes a full data URL , prefix and all: import base64 from openai import OpenAI client = OpenAI () with open ( " photo.png " , " rb " ) as f : b64 = base64 . standard_b64encode ( f . read ()). decode ( " utf-8 " ) resp = client . chat . completions . create ( model = " gpt-4o " , messages = [{ " role " : " user " , " content " : [ { " type " : " text " , " text " : " What ' s in this image? " }, { " type " : " image_url " , " image_url " : { " url " : f " data:image/png;base64, { b64 } " }, }, ], }], ) print ( resp . choices [ 0 ]. message . content ) The literal payload shape: { "type" : "image_url" , "image_url" : { "url" : "data:image/png;base64,<BASE64>" } } Note the data:image/png;base64, is part of the value. Send raw Base64 here and it fails. Anthropic (Claude) Claude uses an image content block with a source object. Here the MIME type is a separate field ( media_type ), and the data field wants raw Base64 — no data: prefix : import base64 import anthropic client = anthropic . Anthropic () with open ( " photo.png " , " rb " ) as f : b64 = base64 . standard_b64encode ( f . read ()). decode ( " utf-8 " ) msg = client . messages . create ( model = " claude-opus-4-8 " , m