I had asked ChatGPT to organize my writing and this is the outcome; it’s language is strange as it refers to itself as the assistant. Anyways, it added more detail then I had initially added and should summarize the main purpose of this request in a concise manner.
I had to remove the images from the tests as I am a new user to the forums.
1. Problem Statement
ChatGPT can generate impressive 2D images, but follow-up edits involving three-dimensional orientation are often unreliable. In my own testing, image generation struggled when asked to rotate a generated object around a specific axis, even when provided with example views from Blender.
This is understandable because a 2D image generator cannot simply rotate the underlying geometry relative to the viewing plane. Once an object exists only as pixels, the model must infer the hidden geometry, perspective, and object structure from the image rather than directly transforming a persistent 3D representation.
This RFC proposes a lightweight 3D intermediary for rigid objects. The goal is not to create a full professional 3D modeling system with UV mapping, rigging, skeletal animation, skin deformation, or production-ready materials. Those features are intentionally out of scope.
Instead, the proposal is for a simple rigid-object geometry layer that can generate basic mesh/submodel structures, preview them in WebGL, and use rendered 2D views as context for further editing or image generation.
2. Proposed Solution
Add a lightweight 3D generation layer that can create simple rigid objects and display them in an embedded WebGL preview window inside ChatGPT.
This system would serve three related purposes:
-
3D model generation
ChatGPT could generate basic OBJ-style geometry with simple per-part materials or colors. -
Interactive inspection
The user could rotate, pan, and zoom the generated object directly in ChatGPT without importing it into Blender or another 3D program just to verify the shape. -
2D image-generation support
ChatGPT could render the 3D object from a controlled angle and use that render as a baseline/reference for 2D image generation or modification.
The key shift is moving from independent image regeneration to a workflow where one object has one persistent structure and many possible rendered views.
A simplified flow:
Prompt
→ geometry plan
→ mesh/submodel generation
→ WebGL preview
→ user rotates/pans/zooms
→ send current view back to chat
→ conversational edit
→ updated geometry or 2D render
→ export
3. WebGL Preview Window
The WebGL viewer should provide basic inspection controls:
-
rotate/orbit
-
pan
-
zoom
-
reset view
-
basic lighting
-
optional wireframe view
-
model download/export
The most important additional feature would be a button similar to the download button on generated images:
Send Current View to Chat
This button would capture the current camera angle and rendered viewport, then insert that image directly into the conversation along with useful camera/rotation metadata.
This would allow a user to rotate the object to the exact area they want to modify and then say:
-
“make this part thicker”
-
“rotate this arm upward”
-
“shorten this handle”
-
“move this section backward”
-
“make the blade wider from this angle”
The assistant would then have both the underlying geometry and the exact 2D viewpoint the user is referring to.
4. Intended Scope
The initial version should focus on rigid objects and rigid submodels.
Suitable examples include:
-
tools
-
weapons
-
furniture
-
vehicles
-
buildings
-
simple props
-
machines
-
modular robots
-
mechanical parts
-
low-poly environmental assets
This is especially useful when the object can be separated into submodels. For example, a robot arm, axe head, wheel, chair leg, or handle can be rotated or resized independently without requiring deformation.
This is not intended to solve character animation. Human characters, animals, skin, cloth, and organic deformation are much harder because the geometry must bend, stretch, and deform naturally. A rigid object pipeline should not attempt to solve that in the MVP.
5. File Format Support
OBJ should be supported as a simple export format because it is widely recognized and easy to inspect, generate, and import into many 3D tools. MTL support would allow simple per-part material colors.
FBX should be considered as a secondary export format because it is common in game-development workflows.
Blender compatibility should be a priority from the beginning. A user should be able to download the generated model and import it into Blender immediately for cleanup or further editing.
For the WebGL viewer itself, the system could use whichever internal representation is most practical. The user-facing output does not need to be limited to the viewer’s internal format.
6. Testing the Current Model
I tested the current model by uploading low-poly OBJ files without their attached MTL files. The files were generated in Blender and renamed generically as mystery OBJ files so the assistant could not rely on filenames.
The tested objects were:
-
low-poly apple
-
low-poly desk/office chair
-
low-poly potted rose plant
The goal was to determine whether ChatGPT could inspect simple geometry, render or reason from multiple views, and infer the object category without material/color information.
Test 1: Low-Poly Apple
The first file was a low-poly apple. The model did not immediately jump to the final answer. It considered several possibilities based on the geometry, including coconut, potato, pear, pumpkin, bird, and duck.
These guesses were not correct, but they were informative. The model recognized that the main body was a rounded ellipsoid shape and that the smaller attached parts might represent a stem, leaf, beak, or other protruding feature. This showed that it was reasoning from the component structure rather than guessing from the filename.
After inspecting the grouped geometry and recognizing the combination of:
-
rounded fruit-like body
-
small stem-like object
-
leaf-like attachment
the assistant concluded:
“It looks like a low-poly apple / round fruit with a stem and leaf.”
This was the correct identification.
The important point is that the model’s intermediate guesses were mostly within the correct shape category. It did not confuse the object with something unrelated like a chair or vehicle. It reasoned toward a fruit-like object by interpreting the geometry and attached subparts.
Test 2: Low-Poly Desk Chair
The second file was a low-poly desk chair / office chair. The model initially guessed that it was a wheelchair.
This was incorrect, but the mistake was explainable. From the initial angle, the lower base and side structures looked similar to wheels or wheelchair supports. The model correctly identified that the object had:
-
a seat-like structure
-
a backrest
-
side supports
-
lower circular or wheel-like elements
After being asked to try again and inspect the object from additional views, it corrected the answer:
“It’s a low-poly office chair / desk chair, likely a swivel chair with armrests and caster wheels.”
This test demonstrated that additional rendered viewpoints can directly improve recognition. The first view led to a plausible but wrong category. Once the model had a clearer understanding of the top and side structure, it correctly identified the object.
This supports the idea that a WebGL viewer with controllable camera angles would be valuable. The user or system could provide a more informative view instead of relying on a single render.
Test 3: Low-Poly Potted Rose Plant
The third file was a low-poly potted rose plant. This turned out to be the most difficult test.
The model initially guessed that the object might be a carrot. It also considered other possibilities such as pineapple, corn on the cob, a firework, or a cartoon bomb. These were incorrect, but they reflected a reasonable interpretation of the first render: the pot and vertical plant structure looked like a tapered object with pieces extending from the top.
After rotating the view and inspecting the model again, the assistant corrected the category and concluded that it was a potted plant.
When asked what type of plant it was, the assistant guessed cactus, succulent, bonsai, bamboo, and also suggested a potted rose stem without visible flowers. The cactus/succulent conclusion was wrong, but the secondary rose guess was close to the correct answer.
The model struggled because the OBJ was provided without the MTL file, so the flower color information was missing. Without red materials, the rose heads appeared as ambiguous geometry. After I provided a Blender screenshot with the proper view and visible red flower heads, the assistant correctly identified the object as potted red roses.
This test was especially useful because it showed the limit of geometry alone. The raw OBJ structure was enough to reach “potted plant,” but not enough to reliably reach “rose.” Once the model had a better 2D rendered view with color and camera context, the specific object became obvious.
This strongly supports the proposed Send Current View to Chat feature. A user-controlled viewport capture can provide exactly the missing visual context that raw geometry or a poor default render may fail to communicate.
7. Test Conclusions
These tests suggest that the current model can already reason about simple 3D geometry when the object is converted into rendered 2D views.
The model does not need full professional 3D understanding to produce useful results. A basic reasoning loop could work as follows:
Generate OBJ
→ render multiple views
→ analyze rendered views
→ compare result against original prompt
→ identify mismatches
→ revise geometry
→ render again
This could allow ChatGPT to improve generated OBJ files through iterative self-review.
The process already works manually, but a built-in system would make it much faster and more reliable. Instead of ChatGPT generating ad-hoc render scripts and previews, the application would already have a consistent renderer, viewport, and model-inspection workflow.
8. Follow-Up Test: Tree Generation
As a follow-up, I asked ChatGPT to generate a simple tree OBJ file and render a preview image.
After generating the first version, I asked it to analyze whether the model matched the original prompt. It concluded that the object was clearly a low-poly broadleaf tree, but also identified potential improvements such as a more intentional silhouette, stronger trunk shape, branches, and leaf variation.
It then generated a second version. After comparing version 1 and version 2, it concluded that version 2 had added useful ideas but produced a worse preview overall. It then generated version 3, attempting to preserve the useful additions while improving readability.
Finally, I asked it to compare version 1 and version 3 while discarding version 2. It concluded:
“V3 improved on V1 without making the same mistake as V2.”
This test demonstrated that ChatGPT can already perform a primitive version of the proposed reasoning loop:
create model
→ render preview
→ inspect result
→ compare to prompt
→ modify geometry
→ re-render
→ compare versions
The results were not perfect, but the loop itself worked.
9. Why This Matters for 2D Image Generation
The same concept could improve 2D image generation.
When a user asks to rotate a generated image, the current system has to infer the object’s 3D structure from pixels. This often fails because the model does not have a persistent geometric source of truth.
With a lightweight 3D intermediary, the workflow could instead be:
Generate or infer simple 3D structure
→ rotate object or submodel in 3D space
→ render from the desired camera angle
→ use the render as the structural baseline
→ apply 2D image generation for style, color, shading, and background
This would be especially useful for rigid or modular objects, such as robots, tools, weapons, vehicles, furniture, or low-poly props.
For example, if a user has a robot image and asks for the arm to rotate upward, the system could treat the arm as a rigid submodel, rotate it mathematically, render the updated pose, and use that render as the basis for the final 2D image.
This would not solve every type of image-editing problem, especially not human body deformation, but it could significantly improve rigid-object consistency.
10. Optional High-Quality Mode
This system would likely add some generation time. That is acceptable if it is reserved for cases where the user explicitly wants higher-quality spatial consistency.
A possible implementation would be:
-
Instant / standard image generation: current 2D-first workflow
-
High-quality spatial mode: geometry-assisted reasoning loop
-
3D object mode: generate/export OBJ/FBX and preview in WebGL
Some code-generation prompts already take several minutes when higher reasoning effort is used. A few additional geometry/render/evaluation loops could be acceptable when the result is significantly more useful.
11. Out of Scope
The MVP should not attempt to include:
-
UV unwrapping
-
advanced textures
-
PBR materials
-
skeletal rigging
-
animation
-
skinning
-
cloth simulation
-
deformation
-
production-ready game assets
-
character posing
Keeping the system focused on rigid object geometry makes the problem much more practical.
12. Expected Benefits
This feature would provide several benefits:
-
better 3D consistency for rigid objects
-
improved follow-up image edits involving rotation or angle changes
-
quick OBJ-style starting points for developers and artists
-
immediate WebGL inspection inside ChatGPT
-
fewer failed image-regeneration attempts
-
easier communication between user and model through viewport capture
-
a foundation for future geometry-aware multimodal workflows
13. Summary
ChatGPT can already generate simple OBJ files, render them, analyze the render, compare the result against a prompt, and revise the geometry. The current process is manual and inconsistent, but the capability is visible.
A built-in lightweight 3D generation layer with a WebGL preview and “Send Current View to Chat” button would make this workflow practical.
The proposal is not to build a professional 3D modeling suite. The proposal is to add a constrained geometric reasoning layer that improves rigid-object generation, previewing, editing, and 2D image consistency.