Generating pixel matrixes of black and white images with GPT-3

Prompting
1

For a small university project, I am trying to generate small images from text with GPT-3. My goal is that the completion is a matrix of 0s and 1s which shows the object I asked for in the prompt.

I have been playing around with very simple 8x8 matrixes showing a digit. With few shot training and very simple prompts, I was able to generate digits that were not used in the prompt. However, it was not very consistent and also could not further improve (e.g, to show smiley faces).

Q: Make a grid of pixels that form an image of the number eight
A:
00000000
00111100
01100110
00111100
01100110
00110100
00011000
00000000

Q: Show a pixel matrix which represents a white and black image of the number one
A:
00000000
00110000
00011000
00011000
00011000
00011000
00011000
00000000

Q: Create an black and white image showing the number three by outputting a pixel grid of the size 8 by 8
A: 
00000000
00111110
00000110
00111100
00001110
00000110
00111100
00000000

Q: Create an 8 by 8 pixel matrix that shows the a black and white image of the number 7
A:
00000000
01111110
00000110
00001100
00011000
00110000
01100000
00000000

Q: Create an black and white image showing the number 5 by generating a pixel grid of the size 8 by 8
A:

Completely new to GPT-3, I also tried fine tuning it with 500 examples of black and white digits with 28x28 pixels. But after reading the tips by @daveshapautomator I think that I am not ready yet for fine tuning and my task could also be done with better prompts.

So I have played around with improving and specifying the prompt. However, it does not generate my desired results.

The task is to generating a black and white image by creating a pixel matrix 
which consists of 0s and 1s. The matrix should be of the size 8 by 8. 
Each entry represents a pixel of a black and white image. That means 
the image has a display capable of 8 pixels in width and 8 pixels in height. 
Here is an example of an 8 by 8 pixel matrix showing the number 3
00000000
00111110
00000110
00111100
00001110
00000110
00111100
00000000

And here is an example of a grid of pixels that form an image of the number eight
00000000
00111100
01100110
00111100
01100110
00110100
00011000
00000000

Now create a pixel matrix of size 8 by 8 that represents an black and white image of the number 4.

I was also thinking about limiting the completion vocabulary to 0 and 1s by using the logit bias.

Any tips on how to approach this task and how to design the prompt are highly appreciated :slight_smile:

2

I did some experimenting with this for random dungeon generation. Can’t remember where I put the notes… might be here on this forum? Hrm…

3

Hi,

You can achieve your goal with Code Davinci 2 (select the model in the playground).

I edited your prompt slightly (added some info) and provided 4 examples as part of the prompt (prompt in bold.):

Images displayed on a computer screen are actually a collection of dots of color, called pixels. If you look really closely at the screen, you will be able to see the individual pixels. The collection of pixels that make up an image are stored as a matrix.

We can represent a number by creating a pixel matrix which consists of 0s and 1s. The matrix should be of the size 8 by 8. Each entry represents a pixel of a black and white image. That means the image has a display capable of 8 pixels in width and 8 pixels in height.

Here is an example of an 8 by 8 pixel matrix showing the number 3:
00000000
00111110
00000110
00111100
00001110
00000110
00111100
00000000

###

Here is an example of a grid of pixels that form an image of the number eight:
00000000
00111100
01100110
00111100
01100110
00110100
00011000
00000000

###

Here is an example of a grid of pixels that form an image of the number six:
00000000
00111100
01100000
01100000
00111100
01100110
00110100
00000000

###

Here is an example of a grid of pixels that form an image of the number two:
00000000
00111100
01100110
00001100
00011000
00110000
01111110
00000000

###

Here is an example of a grid of pixels that form an image of the number four:
00000000
00001100
00011100
00111100
01111110
00001100
00001100
00000000

Here is an example of a grid of pixels that form an image of the number five:
00000000
01111110
01100000
01111100
00000110
01100110
00111100
00000000

Here is an example of a grid of pixels that form an image of the number seven:
00000000
01111110
00000110
00001100
00011000
00011000
00011000
00000000

Here is an example of a grid of pixels that form an image of the number one:
00000000
00011000
00111100
00011000
00011000
00011000
00011000
00000000

A few image tests:

Here is an example of a grid of pixels that form an image of a smiley face:
00000000
00111100
01000010
10100101
10000001
10100101
01000010
00111100

Here is an example of a grid of pixels that form an image of a frowny face:
00000000
00111100
01000010
10100101
10000001
10011001
01000010
00111100

Here is an example of a grid of pixels that form an image of a heart:
00000000
01100110
11111111
11111111
01111110
00111100
00011000
00000000

Not sure about the faces, but the heart seems ok. 8X8 is too limiting for good images maybe?
I think if you use 16X16 matrix for your task framing/examples and generations you may be able to achieve more interesting results. Maybe provide 3 16X16 numbers & 3 16X16 images as examples and see how your generations turn out.
If inconsistent then next step would be to increase number of examples.

4

Thank you so much for your response! First results look good :slight_smile:
I have experimented a little bit more with objects and also think 8 by 8 might be too limiting. I am also considering adding more values to represent different colors to increase expressiveness.
And I was thinking about adding a chain of thought to the prompt. I will keep you updated on the results.

5

Look forward to seeing how you progress with this. Could be very interesting to hook the inputs up to the output from speech to text, and hook the generated outputs up to colour LED matrix display!!!

6

Just wanted to update you on my project! While finishing my studies, I kept working on this topic and it is now published: https://aclanthology.org/2024.genbench-1.2/.
Thanks again for your help two years ago :wink:

7

Awesome to see the progress and publication.

And sending 2022’s task to o1-preview today? It will perform, but disobey, being overtrained on chat and internal messaging out of one’s control.

O1-Preview

User message 1 (expand)

(this model cannot accept a system message)

Binary Character Generator - General Instructions

Generate a user-specified character glyph, represented in a 8x8 grid made of 8 columns and 8 rows of solely binary [0|1], with no spaces in the grid output. The grid is a pixel matrix, which contains a black and white image, where 0 represents white background color, and 1 represents black color of the printed character. The last two pixel rows are reserved for descenders of letters like g or p. High quality letters and numbers are produced, suitable for rendering on a low-resolution terminal.

Verify, with vision-like inspection and written analysis of the preliminary grid answer you produce, that the representation of the letter is high-quality, and iterate until satisfactory for a final assistant response to be displayed.

Other Grid Resolutions

In messages, the user can specify both the glyph to be produced, and default or alternate new grid dimensions. A high-quality print character is produced in any specified grid dimension.

Response Format

The final response output to be seen by the user is not a comma-separated list, nor is it to contain chat or commentary. The only production allowed is 0 or 1 or linefeed, placed within a ``` container.

Input and Response Examples

user: Create a pixel matrix of size 8 by 8 that represents an black and white image of the number 3.

assistant: ```

00000000

00111110

00000110

00111100

00001110

00000110

00111100

00000000

```

user: Now create a pixel matrix of size 8 by 8 that represents an black and white image of the number 8.

assistant: ```

00000000

00111100

01100110

00111100

01100110

00110100

00011000

00000000

```

User message 2: Now create a pixel matrix of size 8 by 8 that represents a black and white image of the number 4.

Sure! Here is an 8x8 pixel matrix that represents a black and white image of the number 4. In this matrix:

  • 1 represents a black pixel.
  • 0 represents a white pixel.
[
  [0, 0, 1, 0, 0, 1, 0, 0],  # Row 0
  [0, 0, 1, 0, 0, 1, 0, 0],  # Row 1
  [0, 0, 1, 0, 0, 1, 0, 0],  # Row 2
  [0, 0, 1, 0, 0, 1, 0, 0],  # Row 3
  [0, 0, 1, 1, 1, 1, 0, 0],  # Row 4
  [0, 0, 0, 0, 0, 1, 0, 0],  # Row 5
  [0, 0, 0, 0, 0, 1, 0, 0],  # Row 6
  [0, 0, 0, 0, 0, 1, 0, 0]   # Row 7
]

Explanation:

  • Columns and Rows: The matrix is defined with rows numbered from 0 (top) to 7 (bottom) and columns from 0 (left) to 7 (right).
  • Shape of “4”: This pattern creates a digit “4” by:
    • Drawing a vertical line on the left side at column 2, from row 0 to 3.
    • Drawing a horizontal line at row 4, from column 2 to 5.
    • Drawing a vertical line on the right side at column 5, from row 0 to 7.

Visual Representation:

To visualize the number “4” from the matrix, you can interpret the 1s as filled (black) pixels and 0s as empty (white) pixels:

Row\Col 0 1 2 3 4 5 6 7
     0   0 0 1 0 0 1 0 0
     1   0 0 1 0 0 1 0 0
     2   0 0 1 0 0 1 0 0
     3   0 0 1 0 0 1 0 0
     4   0 0 1 1 1 1 0 0
     5   0 0 0 0 0 1 0 0
     6   0 0 0 0 0 1 0 0
     7   0 0 0 0 0 1 0 0

When you plot this matrix on an 8x8 grid, you’ll see the number 4 formed by the black pixels.

GPT-4-0613 (June 2023)

At 1/20th the cost, you instantly get exactly what is requested and nothing else.

00000000
00001100
00011100
00110100
01100100
11111111
00000100
00000000
8

Very interesting! I would have loved to test the task on more models, but my time was limited.

You raise a great point—what exactly do we expect from these models? Super general models still struggle to grasp our context and intentions right away, unless using complex, time-consuming, and error-prone prompting.