Anyone know if video ai (Sora, etc) will ultimately allow you the use of story boards in place of prompts for visual direction?

Community
1

This is a question for developers, about Future video ai tools. I can’t seem to find the answer anywhere so I’m asking here. I feel like prompts are pretty limited and counterintuitive if you just want to play around with video using visual direction you’ve already worked out. I understand the value of prompts in cleaning up and editing video. Do developers know if it will be possible ? It would be annoying not being able to use your own visual direction. Not being able to skip the trial and error you encounter with prompts if drawing them is simply easier. Does anybody know if Sora, or any other ai platform allow the option to skip prompts and use however many you need?

2

I think this is already out there, I saw a review on this topic, it seems the Chinese KLING AI does this.

1 Like
3

Can provide a link to the content you read ? Thanks for replying.

4

Image generation LLMs are able to do receive visual input and generate based on it, but that came in second, at first only text prompts were used.

If it doesn’t come with it at launch, I’m sure it is with in the roadmap of features.

In my projection, I believe that this is a matter of “when” will it be able to achieve image generation (at released or later) based on visual inputs and not an “if” it will be able to accept visual inputs (images).

If not, there is a work around it I guess, draw the image, send the image to a vision model to generate a text prompt, feed the prompt to a video generation model.

1 Like
5

In general, you can’t do without text hints, but if I understood your question correctly, you can use the start frame and the end frame of the video you want to link into one when generating the video. You just add the necessary still frames to the tooltips before generation (two pictures as a tooltip). Again: use the last frame of the first video as the start frame for the new generation and the first frame from the second video as the end frame. This way you will have a video that is a continuation of the first and the beginning of the next. This is what KLING AI, LUMA and others do.