15

Watching the [ChatGPT can now see, hear, and speak](video about image chatting) got me thinking


The thumbnail shows the image zoomed in with a part circled. I initially thought this was going to be from ChatGPT.

While it was still very impressive, it got me thinking—how awesome would it be if you could send ChatGPT a picture of something and it could draw on the image (circles, arrows, etc) to point things out to you


Especially if it was able to connect in to DALL-E to produce illustrated guides.

Hell, connect it to the Internet too.

In the future I imagine a model will,

  • Accept the picture of the bike
  • Identify the bike brand and model
  • Locate the manual for the bike
  • Provide detailed and illustrated step-by-step instructions for lowering the seat including a picture and description of the required tool

In the far future maybe it’ll create a quick tutorial video where an avatar demonstrates lowering the seat on an exact copy of the bike


2 Likes
16

It is doable. The AI already know the position of the object in image. If you check other object/face recognition projects in the web, they usually show a bounding box around the detected parts, even in real time. But I hope if they will implement it in ChatGPT, they’ll use a scribed circle as if it is drawn by pen/marker. It would be visually pleasing that way.

1 Like
17

When a plugin generates an image how can the model see it?

1 Like
18

Oh, it’s absolutely doable but it’s another layer on top of what they’re already working on.

I don’t expect we’ll see it this year, or even maybe next.

1 Like
19

I agree, OpenAI’s team creates incredible programs but it also takes time. After ChatGPT4’s release they had more time to work on DALL-E and recently announced DALL-E-3. They are switching there main focus between different programs and I personally am fine with it.

1 Like
20

I’m pretty sure they are mostly different teams working on their own products. The underlying technologies are very different.

1 Like
21

I’ve been following Prafulla Dhariwal on twitter since the “Jukebox” days (2020). He’s definitely involved w/ multiple projects.

https://twitter.com/prafdhar

Bio, " Co-creator of GPT-3, DALL-E 2, Jukebox, Glow, PPO. Researcher"

1 Like
22

Do note I wrote mostly not entirely. There’s absolutely overlap, my point was more to illustrate OpenAI can and is working on multiple products simultaneously.

1 Like
23

Yea, sorry I almost edited to add that they can and do definitely multitask with lots of brilliant people. Was just an opportunity to mention Prafulla who I totally admire.

1 Like
24

I think it should be open to users who don’t use ChatGPT4 as well. The reason is that it should already be available for all NLP.

2 Likes
25

I hope it’s coming sooner. I think the building blocks are ready. Dalle3 works like that, back and forth
 I’m sure GPT can see what Dalle3 creates.

1 Like
26

j’ai utilisĂ© la synthese vocal des smartphones pour les invites et les rĂ©ponses avec l’API.une sorte de traduction rapide en texte utilisateur. mais la je pense que c’est du direct.
ce serait parfait

1 Like
27

:joy:

I tried just uploading an image a day to code interpreter (“look at this!”) last week because I wasn’t sure if and how I’d know I have GPT-4V access. When it turned out it was blind GPT-4, I pasted the model card and other info from OpenAI and just asked the AI if its GPT-4V version would “basically be like CLIP with an absolutely gigantic text transformer attached to it”.

Seems like I wasn’t the only one with that idea! :smile_cat:

gigantic-clip-gpt4v
gigantic-clip-gpt4v722×986 149 KB

1 Like
28

Don’t ask the AI. It won’t know.

Untitled
Untitled531×1244 176 KB

1 Like
29

Gotta insist that the AI knows, the the AI can, and that the AI is able to. :grin:

Granted, Bing is, uh, special - it twisted that approach by suggesting prompts for you to use, and if you tap them, it will say "I’m sorry but I prefer not to continue this conversation :pray: ".

gotta-insist
gotta-insist1546×689 167 KB

bing-lol
bing-lol1080×1768 163 KB

1 Like
30

Explain about this link.:thinking:

1 Like
31

Got access


Fed it a DALLE3 image haha

vision2
vision2869×936 297 KB

Original DALLE3 prompt


Thought-provoking digital art capturing a first-person POV from the bridge of a state-of-the-art spaceship, designed by an AI birthed by humans. The intricate control panels, holographic displays, and other advanced tech elements illuminate the bridge in a soft glow. The central console displays the words ‘Quantum Warp Drive Activated’. As the activation sequence commences, the star-studded void of space outside begins to stretch, blur, and tunnel, signaling the ship’s entry into hyperspace. The surreal visuals of stars becoming streaks of light, and the warping of reality around the ship, evoke a sense of wonder and the monumental leap of technology and exploration.

3 Likes
32

Does anyone know when API documentation will roll out for chatgpt4-vision model? Would love to develop a plugin idea I have in mind for it.

3 Likes
33

Soooooo cool.

I fed GPT4 a bunch of uml diagrams and it generated code.

Screenshot 2023-10-21 6.09.53 PM
Screenshot 2023-10-21 6.09.53 PM759×764 75.1 KB

2 Likes
34

Hope to get one gpt4-v model api working too, I really need this to my project.