OpenAI released ChatGPT-4o with canvas
It’s intended for writing programming codes and writing ; with an interface that contains many features and gives more comfort in creating projects and codes.
Have you tried it? I would like to know your opinion and experience
for more information you can visit : https://openai.com/index/introducing-canvas/
Loving Canvas. Loving the direction coding is moving towards.
A more equal playing field for everyone to express their creativity is always a win in my books.
My biggest issue is that it’s locked behind ChatGPT (The Web App). No file explorer or any of the typical IDE functionality.
But, I feel like IDEs will start to morph into something like this anyways, so. Loving it. The UX is wonderful.
I’ve only been playing with it a short time and it’s definitely an improvement over pasting the code changes over and over into the prompt. I think it will be very useful.
My initial feedback
Cool stuff guys!
There will always be a context window, so maybe instead of trying to do the whole program as one giant file, create modules (whatever that means for your language, files, classes, components, etc.) that can be worked on separately?
I finally had a chance to give it a whirl today.
Kinda slick!
Even pinned a suggestion for me…
I’ve only tested with articles so far. Length seems to be a problem if the article is super long.
Also, it would be nice if it could just edit one section / paragraph from the DOM instead of redoing the entire article?
Still, super slick, imho, and I’m sure it will improve…
I agree, having to refresh the whole thing seems overkill.
I quite like it.
I found that it seems to take “TODO” comments similarly to system prompts/instructions, at least on initial input. It is also less likely to assume that I don’t know how to use the (OpenAI) API, or otherwise defer to outdated knowledge instead of trusting my input (e.g. I can add comments that steer it away from thinking it’s an error).
O1 has a hard time with both of those.
I noticed if it’s a little bit of text it only edited the section I wanted. So, maybe it’s based on length? The article was a bit long as I wanted to test its limits a bit.
Wanted to report back that I did see it edit just a section which was slick. Also, the article had a long table which maybe messed it up?
Hm…how does Canvas handle multiple files that make up a full application?
I’ve found it a fun workflow but ultimately finding severe limitations with how smart the LLM is.
Here’s a remake of an old classic game in Javascript:
I wanted to create the sensation of racing through a forest and the score going up as I progressed without hitting a tree.
Initially it started out very strange with circles (“trees”) going up the screen in two dimensions and the sky being a small vertical block on the side, but we managed to get past that with feedback and soon enough the trees were coming from the horizontal (phew!) horizon and growing in size - great!
I iterated over 15 times to refine the game and was making good progress initially.
Unfortunately I hit a roadblock where the LLM didn’t seem to be able to get over the issue that all the trees were converging on my position, so my movement left and right made no difference.
Despite giving it feedback several times, it was unable to resolve this, even though it read back my feedback and claimed to have solved it.
So yeah, games developers - you are very safe for the time being!
Is this canvas invitation only? Is it a Tier 5+ thing? I saw the link, it says it’s in beta
It’s ChatGPT based and on a specific model selection. I believe you need a paid plus account
(and yes, that confused the hell out of me to begin with)
lol ty, got it loaded up for my son on the other computer…
He uses p5.js rather than Three but a link he was using to display the 3D JS was JS Editor if that helps anyone
Hey interesting tip… ChatGPT really like to make you keep pressing buttons or mouse keys… This is maybe? a good example I think of same-think from website stuff… Tell it to use the keyIsDown() function and not the keyPressed() function for moving 
see if your son can improve this:
I wonder if the LLM would struggle less with another library to force it to “think” differently.
again, though, I think we are hitting limitations here because the LLM simply doesn’t understand the physical world.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>WebGL Forest Game</title>
<style>
body, html {
margin: 0;
overflow: hidden;
background: #87CEEB;
}
canvas {
display: block;
}
#score {
position: absolute;
top: 10px;
left: 10px;
color: white;
font-size: 24px;
font-family: Arial, sans-serif;
}
</style>
</head>
<body>
<div id="score">Score: 0</div>
<canvas id="gameCanvas"></canvas>
<script>
const canvas = document.getElementById("gameCanvas");
const context = canvas.getContext("2d");
let scoreElement = document.getElementById("score");
canvas.width = window.innerWidth;
canvas.height = window.innerHeight;
let playerPosition = 0;
let speed = 1;
let score = 0;
let highScore = 0;
let gameRunning = true;
const trees = [];
function initTree(x, z) {
return { x: x, z: z, initialX: x };
}
function resetGame() {
score = 0;
speed = 1;
gameRunning = true;
playerPosition = 0;
trees.length = 0;
for (let i = 0; i < 150; i++) {
trees.push(initTree(Math.random() * 20 - 10, i * 100 + 500));
}
drawScene();
}
function updateScore() {
scoreElement.textContent = `Score: ${score} High Score: ${highScore}`;
}
function handleMouseMove(event) {
playerPosition = (event.clientX / window.innerWidth) * 2 - 1;
}
function handleKeyDown(event) {
if (event.key === "s" || event.key === "S") {
resetGame();
}
}
window.addEventListener("mousemove", handleMouseMove);
window.addEventListener("keydown", handleKeyDown);
function drawScene() {
context.clearRect(0, 0, canvas.width, canvas.height);
context.fillStyle = "#836953"; // Ground color
context.fillRect(0, canvas.height / 2, canvas.width, canvas.height / 2);
// Update player-relative position and draw trees
trees.forEach(tree => {
tree.z -= speed;
if (tree.z < -50) {
tree.z = 500;
tree.x = Math.random() * 20 - 10;
}
if (Math.abs(tree.z) < 10 && Math.abs(tree.x - playerPosition * 10) < 1) {
gameRunning = false;
if (score > highScore) {
highScore = score;
}
updateScore();
return;
}
});
if (!gameRunning) {
return;
}
// Draw Trees
trees.forEach(tree => {
const treeX = ((tree.x - playerPosition * 10) / 10 + 1) * canvas.width / 2;
const horizonY = canvas.height / 2;
const treeHeight = Math.max(150 / (tree.z / 50 + 1), 1); // Ensure tree height is positive and not too small
const treeBottomY = horizonY + treeHeight / 2;
const treeTopY = horizonY - treeHeight;
context.fillStyle = "green";
context.beginPath();
context.moveTo(treeX, treeTopY);
context.lineTo(treeX - treeHeight / 2, treeBottomY);
context.lineTo(treeX + treeHeight / 2, treeBottomY);
context.closePath();
context.fill();
});
speed += 0.005;
score++;
updateScore();
requestAnimationFrame(drawScene);
}
resetGame();
</script>
</body>
</html>
<!DOCTYPE html>
<html lang="en">
<head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.10.0/p5.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.10.0/addons/p5.sound.min.js"></script>
<link rel="stylesheet" type="text/css" href="style.css" />
<meta charset="utf-8" />
</head>
<body>
<main></main>
<script>
let playerX = 0;
let playerZ = 0;
let playerY = -500;
let speed = 5;
let speedIncrement = 0.01;
let trees = [];
let score = 0;
function setup() {
createCanvas(windowWidth, windowHeight, WEBGL);
resetGame();
createScoreElement();
}
function draw() {
background(200);
// Set up some lighting
directionalLight(255, 255, 255, 0, -1, -1);
ambientLight(150);
// Set camera position to first-person perspective
camera(playerX, playerY, playerZ, playerX, playerY, playerZ - 500, 0, 1, 0);
// Draw an infinite-looking ground plane
push();
fill(100, 200, 100);
translate(0, -400, 0);
rotateX(HALF_PI);
plane(10000, 10000); // Make a large single plane to give the appearance of an infinite ground
pop();
// Update and draw trees
for (let tree of trees) {
tree.z += speed; // Trees move towards the player
if (tree.z > playerZ + 500) {
tree.z = random(-5000, -500); // Reset tree position behind the player
tree.x = playerX + random(-400, 400); // Trees respawn near the player's x position
score++; // Increase score when successfully dodging a tree
updateScoreElement();
speed += speedIncrement; // Increase the speed after dodging a tree
}
// Check for collision with the player
if (abs(tree.x - playerX) < 50 && abs(tree.z - playerZ) < 50) {
resetGame();
return;
}
// Draw tree trunk
push();
translate(tree.x, playerY, tree.z);
fill(139, 69, 19); // Brown color for the trunk
box(30, 200, 30);
pop();
// Draw tree foliage
push();
translate(tree.x, playerY - 150, tree.z);
fill(34, 139, 34); // Green color for foliage
sphere(100); // Use a sphere to represent the foliage
pop();
}
// Handle movement
if (keyIsDown(LEFT_ARROW)) {
playerX -= 10;
}
if (keyIsDown(RIGHT_ARROW)) {
playerX += 10;
}
if (keyIsDown(UP_ARROW)) {
playerZ -= 10;
}
if (keyIsDown(DOWN_ARROW)) {
playerZ += 10;
}
}
function windowResized() {
resizeCanvas(windowWidth, windowHeight);
}
document.addEventListener("click", () => {
if (document.pointerLockElement !== document.body) {
document.body.requestPointerLock();
}
});
function createScoreElement() {
const scoreDiv = document.createElement("div");
scoreDiv.id = "score";
scoreDiv.style.position = "absolute";
scoreDiv.style.top = "10px";
scoreDiv.style.left = "10px";
scoreDiv.style.color = "black";
scoreDiv.style.fontSize = "24px";
scoreDiv.style.fontFamily = "Arial, sans-serif";
document.body.appendChild(scoreDiv);
updateScoreElement();
}
function updateScoreElement() {
const scoreDiv = document.getElementById("score");
if (scoreDiv) {
scoreDiv.textContent = `Score: ${score}`;
}
}
function resetGame() {
playerX = 0;
playerZ = 0;
speed = 5;
score = 0;
trees = [];
for (let i = 0; i < 50; i++) {
let treeX = random(-1000, 1000); // Trees start far away from the player's x position // Trees start near the player's x position
let treeZ = random(-5000, -500); // Trees start behind the player and move towards them
trees.push({ x: treeX, z: treeZ });
}
if (document.getElementById("score")) {
updateScoreElement();
}
}
</script>
</body>
</html>
The problems were that game was 2D, so it was harder to make the trees look like they are moving past the player.
I’ve modified it so that the game is 3D, so it is easier to make the trees look like they are moving past the player.
Sorry for not making the code very similar to your original, as I had tried many runs with. In the end, I could only fix it by making it as similar as possible from scratch.
Thanks for the challenge!
Yeah, I also changed it to 3D.
But the problem remained.
The LLM is bringing reasonable boiler plate to the table, but it is very hard, sometimes impossible, to get the LLM to fix bugs which involve it understanding the spatial arrangements.
I ended up making changes directly and gave up on the LLM refining it any further.
I really don’t think we are anywhere close to AGI 
In this project here I started from scratch
I think the problem is more converting from 2D to 3D… and vise versa… Maybe this in an unnatural use case as there will be little training data for this online
I suspect the LLM simply doesn’t understand 3D space.
<!DOCTYPE html>
<html lang="en">
<head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.10.0/p5.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.10.0/addons/p5.sound.min.js"></script>
<link rel="stylesheet" type="text/css" href="style.css" />
<meta charset="utf-8" />
</head>
<body>
<main></main>
<script>
let angleXY = 0;
let angleXZ = 0;
let angleXW = 0;
let angleYW = 0;
let points = [];
function setup() {
createCanvas(windowWidth, windowHeight, WEBGL);
// Define the points of a 4D tesseract
for (let i = 0; i < 16; i++) {
let x = (i & 1) == 0 ? 1 : -1;
let y = (i & 2) == 0 ? 1 : -1;
let z = (i & 4) == 0 ? 1 : -1;
let w = (i & 8) == 0 ? 1 : -1;
points.push([x, y, z, w]);
}
}
function draw() {
background(200);
rotate3D();
let projected3D = [];
for (let p of points) {
let rotated = rotate4D(p);
let [x, y, z] = projectTo3D(rotated);
projected3D.push([x, y, z]);
}
// Draw edges
stroke(0);
noFill();
for (let i = 0; i < 16; i++) {
for (let j = i + 1; j < 16; j++) {
if (bitCount(i ^ j) == 1) {
let [x1, y1, z1] = projected3D[i];
let [x2, y2, z2] = projected3D[j];
line(x1 * 100, y1 * 100, z1 * 100, x2 * 100, y2 * 100, z2 * 100);
}
}
}
}
function rotate4D([x, y, z, w]) {
let newX, newY, newZ, newW;
// Rotate in XY plane
newX = cos(angleXY) * x - sin(angleXY) * y;
newY = sin(angleXY) * x + cos(angleXY) * y;
x = newX;
y = newY;
// Rotate in XZ plane
newX = cos(angleXZ) * x - sin(angleXZ) * z;
newZ = sin(angleXZ) * x + cos(angleXZ) * z;
x = newX;
z = newZ;
// Rotate in XW plane
newX = cos(angleXW) * x - sin(angleXW) * w;
newW = sin(angleXW) * x + cos(angleXW) * w;
x = newX;
w = newW;
// Rotate in YW plane
newY = cos(angleYW) * y - sin(angleYW) * w;
newW = sin(angleYW) * y + cos(angleYW) * w;
y = newY;
w = newW;
return [x, y, z, w];
}
function projectTo3D([x, y, z, w]) {
let distance = 2;
let factor = distance / (distance - w);
let newX = x * factor;
let newY = y * factor;
let newZ = z * factor;
return [newX, newY, newZ];
}
function rotate3D() {
if (keyIsDown(LEFT_ARROW)) {
angleXY -= 0.04;
}
if (keyIsDown(RIGHT_ARROW)) {
angleXY += 0.04;
}
if (keyIsDown(UP_ARROW)) {
angleXZ -= 0.04;
}
if (keyIsDown(DOWN_ARROW)) {
angleXZ += 0.04;
}
if (keyIsDown(65)) { // 'a'
angleXW -= 0.04;
}
if (keyIsDown(68)) { // 'd'
angleXW += 0.04;
}
if (keyIsDown(87)) { // 'w'
angleYW -= 0.04;
}
if (keyIsDown(83)) { // 's'
angleYW += 0.04;
}
}
function bitCount(n) {
let count = 0;
while (n) {
count++;
n &= n - 1;
}
return count;
}
</script>
</body>
</html>
I’m sure you are right 
. I have to go to bed now but I generated this just now also
It’s a 4D tesseract.
prompt in Canvas:
‘’’
make a simple wasd and arrow key controlled 4D tesseract in p5.js
‘’’
I have just used GPT-4o with canvas to draft an entire patent application. I have written several AI patents before, and I must say that working with 4o+canvas feels like having a personal patent attorney at my disposal. For sure you shouldn’t accept everything that 4o suggested without questioning, but when used properly it can be a huge help.
To give you a sense of how 4o has helped me, I asked 4o to summarize what it does as follows:
Me: review the entire conversation between us, summarize and categorize the types of interaction that we have had. This is for my purpose of reporting the many ways that you have helped me in writing a patent application.
4o+canvas: Here’s a summary and categorization of the interactions we’ve had throughout our conversation. This will help illustrate the variety of ways in which I assisted with drafting and refining your patent application.
These interactions demonstrate the various ways I supported the patent application drafting, focusing on clarity, completeness, professional tone, and strategic structuring to maximize the likelihood of patent approval. Let me know if there are any other parts you’d like to further refine or discuss.
