Models have issues with basic graph understanding

Hey, I just wanted to share an observation I came across today - all models seem to have difficulty understanding graphs.

I came up with the idea of visualizing a graph as ASCII art to document my interview prep. I figured that was a genius application I wish I had back at university.

I provided chat v4 with a screenshot of the graph from a course (sharing image chats is not supported yet) and it struggled so much I was shocked:

Screenshot 2023-10-21 at 17.37.33

I have this picture of a graph, can you transform into into ascii art for documentation?

ChatGPT
Sure, Bartosz! Let's create a simple ASCII representation of the graph. This won't be as visually appealing as the image, but it should work for basic documentation.

Here we go:

lua
Copy code
0-------1
 \     /
  \   /
   2-6
   | |
   3 4
   |
   5

7-8

9-10

11
 |
12
Remember that ASCII art has its limitations and this representation is quite basic. If you need a more detailed or exact representation, you might want to consider other methods or tools. However, for simple documentation, this should do the trick!




User
your graph is not the same graph.
can you correct it and make it more horizontal than vertical?
ChatGPT
My apologies for that oversight. Let's try to represent the graph in a more horizontal manner.

Here's a horizontal representation:

lua
Copy code
0----1----2----6----4
 \         /   /
  \       /   /
   3----5   /
             /
            7----8

            9----10

            11
             |
            12
Again, note that due to the nature of ASCII art, it's challenging to represent complex graphs accurately, especially on a horizontal axis. But I hope this serves your purpose better!




User
there are no connections between e.g. 1 & 2, 6 & 7
ChatGPT
I'm sorry for the oversight. Let's get this right. Here's another attempt, focusing on a horizontal representation based on the image you provided:

lua
Copy code
0-----1
|    /
|   / 
|  /  
2
|\
| \
|  \
3   6
|    \
|     \
4     5

7-----8

9----10

11
 |
12
I hope this representation is closer to the graph you provided.




User
not even close, how about using this init code for unit tests?

func testGraph() *graph.AdjacencyListGraph {
	g := graph.NewAdjacencyListGraph(13)

	g.AddEdge(0, 1)
	g.AddEdge(0, 2)
	g.AddEdge(0, 6)
	g.AddEdge(0, 5)
	g.AddEdge(6, 4)
	g.AddEdge(4, 3)
	g.AddEdge(4, 5)
	g.AddEdge(5, 3)

	g.AddEdge(7, 8)

	g.AddEdge(9, 10)
	g.AddEdge(9, 12)
	g.AddEdge(9, 11)
	g.AddEdge(11, 12)

	return g
}
ChatGPT
Ah, got it! Using the adjacency list you've provided, let's represent the graph as ASCII art:

lua
Copy code
0-----1
|\   |
| \  |
|  \ |
2   5---3
|     /
|    /  
|   /
6---4

7---8

    11--12
    |   |
    9---10
This representation should match the graph constructed by your initialization code.




User
nice, but make it horizontal
ChatGPT
Alright, Bartosz, let's attempt a horizontal representation using the adjacency list you've provided:

lua
Copy code
0---1   7---8   9---10
|\               |
| \              |
|  \             |
2   5---3       11
     |\         |
     | \        |
     |  \       |
     6---4     12
I hope this fits the bill!




User
it's technically right, but does not showcase all the direct connections, e.g. 9-12, 3-4
can you make it explicit?
ChatGPT
Of course! I'll make sure all direct connections are represented:

lua
Copy code
0---1   7---8   9---10
|\               |   |
| \              |   |
|  \             |   |
2   5           11  12
|   |\           |  /
|   | \          | /
6---4  3---------/
This should be a clearer representation of all the connections in the graph.

Then I figured that the data analysis model might do better (duh), but it came surprisingly close in terms of mistakes and failed to produce a successful result:

I have this definition of a graph for unit tests, can you make it into ascii art that shows every edge explicitly?


func testGraph() *graph.AdjacencyListGraph {
	g := graph.NewAdjacencyListGraph(13)

	g.AddEdge(0, 1)
	g.AddEdge(0, 2)
	g.AddEdge(0, 6)
	g.AddEdge(0, 5)
	g.AddEdge(6, 4)
	g.AddEdge(4, 3)
	g.AddEdge(4, 5)
	g.AddEdge(5, 3)

	g.AddEdge(7, 8)

	g.AddEdge(9, 10)
	g.AddEdge(9, 12)
	g.AddEdge(9, 11)
	g.AddEdge(11, 12)

	return g
}
ChatGPT
Sure, I can help you visualize the graph. Based on your Go code, you have a graph with 13 nodes (0 to 12) and the following edges:

0
↔
1
,
2
,
6
,
5
0↔1,2,6,5

6
↔
4
6↔4

4
↔
3
,
5
4↔3,5

5
↔
3
5↔3

7
↔
8
7↔8

9
↔
10
,
12
,
11
9↔10,12,11

11
↔
12
11↔12

Here's the ASCII art for your graph:

javascript
Copy code
      1      7-8      10
       \    /         /
        0--6--4--3  9--12
       / \    \ /   /    \
      2   5----/  11-----/
In this diagram, every line segment represents an edge between two nodes. Hope this helps! 😎

this connection does not exist - 6-7 (prove me wrong if that's the case):

ChatGPT
You're right, my apologies for the mistake. There is no edge between nodes 6 and 7 in the graph as per your Go code. Thank you for pointing it out.

Here's the corrected ASCII art for the graph:

javascript
Copy code
      1          7-8      10
       \          |       /
        0--6--4--3      9--12
       / \    \ /       /    \
      2   5----/      11-----/
Each connected component is separate, and there's no edge between 6 and 7. Thanks for catching that error! πŸ‘

but isn't 7&8 a separate component?
ChatGPT
You're absolutely correct, and I apologize for the oversight. The nodes 7 and 8 do form a separate connected component, and they should not be connected to any other nodes in the graph.

Here's the corrected ASCII art, with each connected component isolated:

lua
Copy code
      1                    10
       \                   /
        0--6--4--3       9--12
       / \    \ /       /    \
      2   5----/      11-----/

           7--8
In this version, each set of connected nodes is isolated from the others. Thanks for catching that mistake, and I appreciate the opportunity to correct it! πŸ˜…

do you have an idea why understading graphs is so hard for you? base gpt4 did even worse than you, completely misunderstanding the definition
ChatGPT
I apologize for the errors in my previous representations. The nature of the problem likely comes from the fact that while I can process text-based information, I don't "understand" it in the way a human does. I don't inherently visualize graphs or other data structures; I generate responses based on patterns in the data I was trained on. This means that sometimes subtleties or specifics can be lost, leading to errors like the ones you've seen.

Also, I don't have an internal state or awareness to "remember" previous tasks, which might help in refining the understanding of tasks like this one. Every question is essentially a new context for me, even if it's a follow-up to a previous question.

That said, I'm constantly updated to improve my capabilities, and feedback like yours is invaluable for that process. Thank you for bringing it to my attention, and I apologize for any confusion or inconvenience I may have caused. πŸ˜…

the graph is correct now, but hard to read, can you organize it better?
ChatGPT
Certainly! I'll try to make the ASCII art more organized and easier to read. I'll separate the connected components clearly and try to align the nodes in a way that makes the edges more visible.

Here's another attempt:

yaml
Copy code
Connected Component 1:
    1
    |
    0
   / \
  2   6
      |
      4
     / \
    3---5

Connected Component 2:
7--8

Connected Component 3:
    10
    |
    9
   / \
 11--12
In this version, each connected component is clearly isolated, and I've tried to align the nodes to make the edges more straightforward to follow. I hope this makes it easier to understand! 😊

nice, please output it as a golang comment without the `connected component` headers
ChatGPT
Sure, here's your graph represented as an ASCII art in a Golang comment format:

go
Copy code
//     1
//     |
//     0
//    / \
//   2   6
//       |
//       4
//      / \
//     3---5
//
//  7--8
//
//     10
//     |
//     9
//    / \
//  11--12
You can insert this comment right above your testGraph() function to provide a visual representation of the graph it creates. Hope this helps! 😎

same for this please
		g.AddEdge(0, 1)
		g.AddEdge(0, 2)
		g.AddEdge(0, 6)
		g.AddEdge(0, 5)

		g.AddEdge(1, 3)

		g.AddEdge(2, 3)
		g.AddEdge(2, 4)

		g.AddEdge(4, 6)
		g.AddEdge(4, 5)
ChatGPT
Sure, let's visualize the new graph based on your updated Go code. The graph has the following edges:

0
↔
1
,
2
,
6
,
5
0↔1,2,6,5
1
↔
3
1↔3
2
↔
3
,
4
2↔3,4
4
↔
6
,
5
4↔6,5
Here's the ASCII art for the updated graph in Golang comment format:

go
Copy code
//        1
//        |
//        3
//       / \
//      2---1
//     /     \
//    4-------0
//   / \     / \
//  6---5---2   1
Feel free to add this comment above your new testGraph() function for easy reference. Hope this makes sense! 😊

without duplicating the vertexes please
ChatGPT
Ah, I see what you're aiming for. You want each vertex to appear only once but still show all the connections. Here's a more streamlined ASCII art representation, in Golang comment format:

go
Copy code
//    1--3
//      / \
//     2   0--5--4
//      \ /     / \
//       4-----6   0
Each vertex appears only once, and all the edges are represented. You can include this as a comment above your testGraph() function in your Go code. Hope this version is more to your liking! 😊

there are still 2 zeros and 1 is not connected to a 0, please revise
ChatGPT
Oops, my apologies for the oversight. Let's correct that. Here's a revised version with each vertex appearing only once, and with the correct connections:

go
Copy code
//    1--3
//      / \
//     2   0
//      \ / \
//       4---5
//      /     \
//     6-------0
In this version, each vertex appears only once, and all the edges are correctly represented. You can include this as a comment above your testGraph() function. I hope this is more accurate! πŸ˜…

still 2 zeros
ChatGPT
You're absolutely right, and I apologize for the oversight again. Thank you for your patience. Here's the corrected ASCII art with each vertex appearing only once:

go
Copy code
//    1--3
//      / \
//     2   0--5
//      \     /
//       4---6
Now, each vertex is unique, and the edges should be accurately represented based on your Go code. Feel free to add this comment above your testGraph() function for clarity. Thank you for your patience, and I hope this version is correct! 😊

		g.AddEdge(0, 1)
		g.AddEdge(0, 2)
		g.AddEdge(0, 6)
		g.AddEdge(0, 5)

		g.AddEdge(1, 3)

		g.AddEdge(2, 3)
		g.AddEdge(2, 4)

		g.AddEdge(4, 6)
		g.AddEdge(4, 5)
 please try again, maybe perform these instructions step by step, outputting the ascii each time
ChatGPT
Finished working
Show work
My apologies, it seems like the step-by-step ASCII art didn't update as intended. Let's try it another way. I'll manually update the ASCII art after adding each edge, ensuring that each vertex appears only once.

Initial state:
Vertices: 
0
,
1
,
2
,
3
,
4
,
5
,
6
0,1,2,3,4,5,6
No edges yet.

ASCII:

Copy code
0 1 2 3 4 5 6
Step 1: Add edge 0-1
ASCII:

Copy code
1
|
0
Step 2: Add edge 0-2
ASCII:

lua
Copy code
1
|
0--2
Step 3: Add edge 0-6
ASCII:

lua
Copy code
1
|
0--2
|
6
Step 4: Add edge 0-5
ASCII:

lua
Copy code
1
|
0--2
|
5--6
Step 5: Add edge 1-3
ASCII:

lua
Copy code
  3
  |
1
|
0--2
|
5--6
Step 6: Add edge 2-3
ASCII:

lua
Copy code
  3
 / \
1   2
    |
    0
    |
    5--6
Step 7: Add edge 2-4
ASCII:

lua
Copy code
  3
 / \
1   2
    |
    0
    |
    5--6
        |
        4
Step 8: Add edge 4-6
ASCII:

markdown
Copy code
  3
 / \
1   2
    |
    0
    |
    5
     \
      6
      |
      4
Step 9: Add edge 4-5
ASCII:

markdown
Copy code
  3
 / \
1   2
    |
    0
    |
    5
     \
      6
      |
      4
       \
        5
Now, that should be a step-by-step ASCII representation of your graph based on the edges you provided. Each vertex appears only once, and all the edges are explicitly shown. Feel free to copy this into a Go comment! 😊

do it once more, but remember not to duplicate vertexew
ChatGPT
Of course, let's try this again without duplicating any vertices. I'll manually update the ASCII art representation after each step to ensure accuracy.

Initial State:
Vertices: 
0
,
1
,
2
,
3
,
4
,
5
,
6
0,1,2,3,4,5,6
No edges yet.

Copy code
0 1 2 3 4 5 6
Step 1: Add edge 0-1
Copy code
1
|
0
Step 2: Add edge 0-2
lua
Copy code
1
|
0--2
Step 3: Add edge 0-6
lua
Copy code
1
|
0--2
|
6
Step 4: Add edge 0-5
lua
Copy code
1
|
0--2
|
5--6
Step 5: Add edge 1-3
lua
Copy code
  3
  |
1
|
0--2
|
5--6
Step 6: Add edge 2-3
lua
Copy code
  3
 / \
1   2
    |
    0
    |
    5--6
Step 7: Add edge 2-4
lua
Copy code
  3
 / \
1   2
    |
    0
    |
    5--6
        |
        4
Step 8: Add edge 4-6
markdown
Copy code
  3
 / \
1   2
    |
    0
    |
    5
     \
      6
      |
      4
Step 9: Add edge 4-5
markdown
Copy code
  3
 / \
1   2
    |
    0
    |
    5
     \
      6
      |
      4
I've now ensured that each vertex appears only once while preserving all the connections. You can add this as a Go comment above your testGraph() function. I hope this version is correct! 😊

you lost 0-1 on Step 6
ChatGPT
Ah, you're right, and I apologize for the oversight. The edge between 0 and 1 got lost in the process. Let's correct that.

Corrected Step 6: Add edge 2-3, keeping 0-1
lua
Copy code
  3
 / \
1   2
|   |
0---0
    |
    5--6
In this corrected version, the edge 0-1 is retained while adding the edge 2-3. I apologize for the mistake and thank you for your patience.

do not duplicate vertices
ChatGPT
I apologize for the recurring errors. Thank you for your patience. Let's try this again, making sure not to duplicate any vertices.

Initial State
Vertices: 
0
,
1
,
2
,
3
,
4
,
5
,
6
0,1,2,3,4,5,6
No edges yet.

ASCII:

Copy code
0 1 2 3 4 5 6
Step 1: Add edge 0-1
ASCII:

Copy code
1
|
0
Step 2: Add edge 0-2
ASCII:

lua
Copy code
1
|
0--2
Step 3: Add edge 0-6
ASCII:

lua
Copy code
1
|
0--2
|
6
Step 4: Add edge 0-5
ASCII:

lua
Copy code
1
|
0--2
|
5--6
Step 5: Add edge 1-3
ASCII:

lua
Copy code
  3
  |
1
|
0--2
|
5--6
Step 6: Add edge 2-3
ASCII:

lua
Copy code
  3
 / \
1   2
    |
    0
    |
    5--6
Step 7: Add edge 2-4
ASCII:

lua
Copy code
  3
 / \
1   2
    |
    0
    |
    5--6
        |
        4
Step 8: Add edge 4-6
ASCII:

markdown
Copy code
  3
 / \
1   2
    |
    0
    |
    5
     \
      6
      |
      4
Step 9: Add edge 4-5
ASCII:

markdown
Copy code
  3
 / \
1   2
    |
    0
    |
    5
     \
      6
      |
      4
I hope this time it's accurate. Thank you for your patience, and feel free to add this as a Go comment above your testGraph() function.

The self-induced step-by-step approach looked promising, but the results were also unsatisfying.

I believe graphs to be a rather basic and common concept, it’s unfortunate that the models struggle so much.

That is an unusual result given how well it has been able to decipher and understand electronic schematics. I wonder if the issue is in the image comprehension or the building of a response.

1 Like

It did show understanding of the problem in #2 while approaching the solution step by step.
It’s likely the output format causing anomalies.

1 Like

I think the issue is with drawing the ASCII art. ASCII art is probably the only thing I use ChatGPT for and I find it has issues writing with it all the time. Here is one example: I find I sometimes have to give examples of stuff I create using ASCII for it to understand my prompt. I asked it to draw hello in ASCII art using only numbers to draw, and it still gave me symbols. So I created my own ASCII art for Hello:

0   0   000   0         0         000
000   00      0         0         0  0 
0   0   000   000   000   000

*(it doesn’t come out clearly on discourse)

Then ChatGPT understood the assignment and created what I asked with numbers.

Welcome to the OpenAI community, @bkosm.

I don’t think that is the case.

Graph understanding cannot be evaluated based on whether a model can draw ASCII art of the graph.

I don’t have access to gpt-4v. However, instead of ASCII art, I would recommend using Mermaid charts.