Fantastic GPT-40 but...where are the docs?

API
,
1

https://openai.com/api/pricing/ has a link for docs on GPT-4o and it points to the link https://platform.openai.com/docs/models/overview which has no mention of GPT-4o

Is this already available on API? I guess so because the playground shows it when I am logged in

Anyone tried it yet?

3 Likes
2

https://platform.openai.com/docs/models/gpt-4o

:slight_smile:

4 Likes
3

Example script to use it:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
  model="gpt-4o",
  messages=[
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What’s in this image?"},
        {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
          },
        },
      ],
    }
  ],
  max_tokens=300,
)

print(response.choices[0])
3 Likes
4

The first thing is that this model uses a different token encoder. If you were counting, you’ll have to grab tiktoken released 15 minutes ago.

MODEL_PREFIX_TO_ENCODING: dict[str, str] = {
    # chat
    "gpt-4o-": "o200k_base",  # e.g., gpt-4o-2024-05-13
    "gpt-4-": "cl100k_base",  # e.g., gpt-4-0314, etc., plus gpt-4-32k

Found that max_tokens and usage of 256 is counted at 258-268 by cl100k-base of my script, a hair more efficient on English:

However 512 gpt-4o tokens of Japanese is 702 tokens of cl100k (GPT-4) output.

Day 0 speed (110tps is about where gpt-3.5-turbo-instruct maxed out hours after release)

For 3 trials of gpt-4-0125-preview @ 2024-05-13 11:03AM:

Stat Minimum Maximum Average
stream rate Min: 24.4 Max: 31.8 Avg: 27.267
latency (s) Min: 0.78 Max: 1.103 Avg: 0.900
total response (s) Min: 8.848 Max: 11.2203 Avg: 10.373
total rate Min: 22.816 Max: 28.933 Avg: 24.972
response tokens Min: 256 Max: 256 Avg: 256.000

For 3 trials of gpt-4o @ 2024-05-13 11:03AM:

Stat Minimum Maximum Average
stream rate Min: 107.6 Max: 112.9 Avg: 110.200
latency (s) Min: 0.3701 Max: 0.525 Avg: 0.434
total response (s) Min: 2.6818 Max: 2.8598 Avg: 2.773
total rate Min: 90.216 Max: 96.204 Avg: 93.342
response tokens Min: 258 Max: 260 Avg: 258.667

The response is in about 5% less streaming chunks.

3 Likes
5

Nice chart _j thanks!

I just spent the last few weeks fine-tuning GPT 35 and got some awesome speeds but comparing to gpt-4o now makes me feel like I wasted my time! I hope we can fine tune gpt-4o ASAP. very nice though.

1 Like
6

Do you know how to use mp3 or mp4 files with this model?

7

As per the docs, it doesn’t seem to have audio input at this time

image
image823×144 9.24 KB

3 Likes
8

Oh, so this is in fact the better, fastest and cheaper gpt-4-vision? There is no voice as input or output.

9

So, in addition to the image URl or other input given in a reply, the CURL request is the same?

I await the docs for using audio in the API: that will be a killer (not so keen on video yet, for my use cases)

Anyone has any idea when this will be supported?

10

there’s an official staff post here:

curl looks the same. As usual, you can go on the playground and click on show code button top right (</>) https://platform.openai.com/playground/chat?lang=curl&models=gpt-4o

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
  "model": "gpt-4o",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "hello hello"
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "what do you see?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAm4AAAKoCAIAAABBe+TtAAAgAElEQVR4nOy9aZQdx3UmeG9ELm9/9WrHUgRAkFgIcF9ESqQkSqI2jyV2Wx55kTdZrfacPnbPOTPzY053e850e2RP+0zL0/Z42j0+li27u7WvlERSpGRSIglxAQkQAAmAWGxH0FmTaIyDmsYRi2220AqFar7EhWSnmeF0UR7wrn+z4iPvHEEzfffHO1Wq3VauxnNfZ20yB7r9mWcCl9YP+0CUEyYUfGdp0beHPkyHFZyPNKLw6TRdrr9TzPq9Vqtm1nYz6XzLxKqTNnzszPz69bt254eJgPZlMhOfIlWxKBA44A4OzZs9PT08VicefOnbA4psbM8iZphzmYyyxkqTrbH1NQyZisAWCNvElO5bx25fpMqBEHNiulXNddY52xGjg4i4g40ZbXOnyQ+Zjv2vM8y7Kq1eoaTTE3CyF4JzvLsrrdrsltzZEjR46L4mdUlV56aj8TGIvL6enpbrcbRVG9Xl9tMzIObTVF3lkn8V6eq7XPiopptb+/XwhRLpeXmCIRkXvCgpXTQgwd8r6n2Q4bjjdlB7J3vYaj17Ksaxq/yhUKjceUU2mvoB229JqMUg4Hi+OYVwylUombLRQKa7fD8pStx6a4/yWq2xw5cuRg/MxRqakla46sXca91WqZTb8p3Qtl7ZgjAKhUKqy3WCqt0R/TJv8v705aqVQ8z+MYV3Mmn1MoFNgyyS7VTqcDAMVicUk1A2ZoSAXoErpae4eZ5WuLq2jzZNMuEzxX0r+y9pdQPg87H8TMDjMmTHcNwuZMYnOClPKiBJwjR44cWfzMUenl5pCwHmX33tDQEO8Ms2QXl+Wo1WpCiEqlYnh3bUO6qY9vTpueni4UCpVKhWOR+KAhD+OzDMOQJW/Wd8isz98yFYOllKbPa9x+HMdLSgLRlZbvXw1KKa4wbOy6vCH5ZTVihiJLmabbURRduHBBa93f318sFrPlkJaAv9jpdNgRzrvoXMqeATly5Mhh8DNHpcsF2drM2tfXZxyZpVKJDZIsbde4Cm/lfSn0Y0KcjPOPW2YpzK/Z7Bqu5MCV7qWUAwMDbCk1SSNL7ii7nfilLCCWl63n7l1FKjXps1ybIo7jMAwvl0qZ8qMo8n0fAMwgAIBt2xwAzI5PtoRzAeHlsG2bbbxcHCM37ebIkeMK8DNHpYxL95VyDgwbAH3fNzmOa9R6dRyHXarMpmEYXrSyLqbVaC3LYrNnrVZzHIe1ptnCkzuTzWrl9nu9Hn/REJ6x8ZqUVnYuZnenWXt8TKXcqx6YxrupM3Xx6xVYj3m5YPie35g1RKlUklIyR3qeFwRBX1/fiu0opdgGzmWP+GB2XZIjR44cF8XPHJUuqR/L8UErOggZfDLP+KYKj23bqxEMp/z3ej2ttcnxWHv/MsiUDzRHGo0GEydXjjVV+swUb/I32u32xMTE6Ogol8FjVo6iiLmQixdyBQPWspCxJy8HVwWCTPWlNayjVwbLskqlEnePe3IFEbysmzlwFzJVkc34mM12+FmsEfaVLb9seH2Nqho5cuTIsQTXnEoJjE9RXLNkPQ0gLvFVKa1UFMdaqQhASIlEjpRoWc6K52utLYu1nXYcB5HSKyKABpCZVwGpHGQJyOR3KfubsjZiIcUFlbJTP0cFM1v4vs9BMUopPsL0yRup8r6qzPrGici+Se7SRYcyq2u5hatOpTwgZjVzZRG8vDMdLC5/wQfZf8zivlQqsW97jaZ49Hj9IYTguOIr6NKVgQAQwIxvntCaI8ebEdc2r5RAhzpkWWZJB7SOIiWltC0bAeIIojAUQriuxdNJGEaOawMuiugJgyAMw8oqqYGkNSIl5IeG0rTX8wE0ERIpAGFei8Wy1rFSpHXMG3ZZliMlAkogRYQoIGmNUKkIURIpRIlI/J6/67jFlah3VQS+v6hm+jJWU3HMlMlPpGCEGj+g1VgwzTcVy9maiDTHDy/S3CYwxxzMZMdqIrJdCwDiKGq3257nAYBlWVw2AVZJpV0OIaXpW/ZbcaSTQUAEk7eD2O10TJIPf0tKiUKoWC/pZ/I+HWzSib2aj/ie12rPx3Fcr9fLlQoAaI7ZFmLFkYzCMKvCzXsuEMFdNbWuELHX61FaTN8ElLG2xsWbqmqtgXg3m8QSwHZj23WESLgziqjre6jJLRUtSxCBLXI2zZHjzYdrrkoREQQSkSZNQBoAtSaCZrPl2IVS0UlmDgJAcFx7fm4OUAu0pIVS2EICkLBt2+v6gBpILHkVIqExQ6iIEkAXS6XVtakEEEAqS8AqjrWOiRCRhLCYMuM4lNIG0ABqCZUuvyIz1pLturLveaZmvoTU29dqtYwdlU/j6TgbmwOpkuOkTMjM+waVSsU4CxeMlhpdtwgr+TtNTVpDcogIgN1uV1PMdYV46hfpbthLbsdECK/43EW6hYshJ/4KB/gw8WSJ1vQ5e1wQmdUeHzdX970AMlbo1FAPbsEuxkWttamDv7DIWIn+7VUEaGlZGX1G/RIFqymdT8Kyk78yrZTvh935eRXT4HADAEBgqVhizewFqt1qjg71X1L7OXLk+GnCtaZSEccaAIWUAi0EAhEDACAVCo7rJrNS0OsFQeC6rlsq9aVbQEOiWIhnQK1oRWpERECNIAE18CsJQN1udvg9CspSb8EtAWogIlBaKaUjFZOmuFbtAwGkEQUJtDQAkAYQhUIZUINGSNshBZpi06ZpH0ECgLSNXFrEXiQQEYFYxCRBPYgohQ0AJJIyRoYypdl9jLn5MqNgEvogIoIsmWXZK5VWmNFBolKtslXAsm02w7LNc4XFASIABL6/YgeMk3WJMBVCCAlC8rfRDJLSEQrSBAhJjXvUKISIo0UJPIs03yIpKXgN1Gq1ut0u/3QKhYKJE26320YXcvqpcZEaPl6y4MgeN5+aJYgxMPCKJwgCs/rh3WbYgm0..."
          }
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "This image appears to be an illustration containing several eyes grouped together in a circular, cross-like pattern. Each eye has a black pupil with a white background. \n\nAt the top-left corner, a speech bubble contains the following dialogue:\n\"It looks like you are afraid\"\n\"BE NOT AFRAID\" (in larger, less neat text)\n\nBelow this dialogue, there are two additional speech bubbles:\n\"I AM AFRAID\"\n\"I'\''M NOT AFRAID\"\n\nThe entire image is set against a light background, and there are several black splatters around the edges of the illustration. The eyes are connected, giving the impression of an abstract creature or object."
        }
      ]
    }
  ],
  "temperature": 1,
  "max_tokens": 256,
  "top_p": 1,
  "frequency_penalty": 0,
  "presence_penalty": 0
}'
2 Likes