Introduction
Computational storyboarding builds upon traditional storyboarding techniques, combining elements from screenplays, storyboards, functions, diagrams, and animation.
Computational storyboards are intended to be of use as input for generative artificial-intelligence systems to create longer-form output video.
A motivating use case is simplifying the creation of educational videos, e.g., lecture videos. With computational storyboards, content creators could describe single-character stories where the main characters were tutors instructing audiences with respect to provided subject matter, perhaps utilizing boards or screens displaying synchronized multimedia content from textbooks, encyclopedia articles, or slideshow presentations.
Screenplays
A screenplay is a form of narration in which the movements, actions, expressions, and dialogue of characters are described in a certain format. Visual and cinematographic cues may also be given, as well as scene descriptions and changes.
Storyboards
A storyboard is an organization technique consisting of illustrations or images displayed in sequences for purposes including the pre-visualization of motion pictures, animations, motion graphics, or interactive media sequences.
Computational storyboards can be much more than sequences of thumbnail images with accompanying text. Thumbnails could additionally provide information about content layering, audio and sound effects, camera shots, character shots, transitions between scenes, and so forth.
Thumbnails could refer to multimedia resources to benefit generative artificial-intelligence systems, e.g., reference materials with respect to characters, settings, props, and style.
Per the motivating use case of educational video, another type of referenced multimedia resource would be that content intended to be synchronized and placed onto one or more surfaces in generated video content.
Web of Computational Storyboards
Nodes in diagrammatic computational storyboards could refer to diagrammatic resources by URLs, weaving a Web of diagrams. End-users could click on these nodes to expand them, loading contents from URL-addressable resources into diagram contexts.
Wiki of Computational Storyboards
Computational storyboard diagrams could be collaboratively editable, enabling wiki platforms.
Functions
Some nodes in computational storyboard diagrams could contain or refer to sub-diagrams by URLs while invoking them, passing arguments to them. Functions would enable modularity and the reuse of storyboard content.
With functions, scenes’ characters, settings, props, actions, dialogue, and properties of these could be parameterized.
Arguments provided to invoked functions could be either multimedia content, text, or structured objects. Functions’ arguments and variables in them could be used to create the prompts provided to generative artificial-intelligence systems, e.g., the prompts with which to generate the thumbnail images.
Computational storyboarding functions could be annotated with metadata in the form of example argument sequences for content creators to select from so that they could have generated visual contents available while designing.
Control Flow
With respect to computational storyboarding diagrams, there are two varieties of control-flow constructs to consider.
A first variety of control-flow construct would route execution using input arguments and variables to paths of subsequent thumbnails.
A second variety of control-flow construct would result in branching or interactive video output, with routes or paths to be selected during playback. Generated interactive video content could interface with playback environments, e.g., in Web browsers, to provide viewers with features. Uses of interactive video include providing viewers with navigational menus.
Marking Points and Intervals of Time
Markers, resembling keykodes or timecodes, could be placed between thumbnails in computational storyboard diagrams, e.g., to use when referencing points or intervals of continuous shots or scenes. Alternatively, some or all thumbnails could be selected to serve as referenceable markers, keykodes, or timecodes in resultant video.
Generating Thumbnail Images
As considered, computational storyboards’ thumbnail images would be created by generative artificial-intelligence systems from provided dynamic prompts. These prompts could be varied including by means of using functions’ input arguments and variables.
Generating Video from Computational Storyboards
A goal is that generative artificial-intelligence systems would be able to process computational storyboards to create longer-form video content.
Towards achieving this, in addition to dynamic thumbnail sequences, computational storyboards could provide useful supplemental materials for generative artificial-intelligence systems, e.g., text or structured-knowledge notes about directing, cinematography, and characters or acting.
Generated videos could utilize multiple tracks to enable features including interoperability with viewers’ client-side artificial-intelligence systems. Embedded transcripts or captions, for example, alongside accompanying metadata track items, could be sent to asssistants for these systems to be able to answer questions about videos’ contents.
Processing Video into Computational Storyboards
In theory, existing video content could be processed into computational storyboards.
Conclusion
Envisioned computational storyboards build on traditional storyboarding techniques while intending to enable generative artificial-intelligence systems to create longer-form output video, e.g., educational video.
Happy holidays! Any thoughts on these ideas?