When will it be possible to create physics and or biology images that is for illustrating text books and exams? Anyone knows? Or anyone knows how to do that already?
1 Like
The AI creates images of fantastic fantasy.
It is not suitable for education or documentation by content.
You might be able to make simple well-described human-supervised diagrams such as levers in physics.
Outside of the DALL-E image maker, I ask the AI about what “advanced data analysis”, a python code interpreter, can do for diagrams itself…
The following vector graphics libraries are available in the Python environment:
matplotlibplotlygraphvizsvgwriteGiven these options,
matplotlibis the most appropriate for creating a free body diagram of an airfoil with force vectors. It’s widely used for scientific and technical visualizations. I’ll now create a demonstration of such a diagram usingmatplotlib.
And having AI produce such a diagram…its abilities are underwhelming.
Metplotlib is definitely my go-to when it comes to math graphs. It’s da bomb.
But since I started doing physics and biology related topics I started to cry… lol
Anyway, any updates on this would be super helpful. I will keep searching on hugging face and other platforms for any updates too. Much appreciated you guys!
1 Like
That is an excellent question and one I have an interest.
Have you read the paper on how DALL-E 3 creates images?
“Improving Image Generation with Better Captions” (PDF) by James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra,Prafulla Dhariwal, Casey Chu, Yunxin Jiao, Aditya Ramesh
The paper elucidates the constraints and opportunities in generating these images.
Short answer: No.
My thoughts on this are that with an image generation prompt one should do something similar to a multishot text prompt but include images, I have not looked for such but it is on my list. I also think that the technical images are distinct enough from the types of images normally created with current image generators that that the training set needs to be entirely different or very selective.
Please let me know if you have any progress on technical, physics or biology images/illustrations for exams and text book. I’ll also share on this forum if I found anything, much appreciated!
First off creating images is not high on my list but it is on my list so I am currently not putting a lot of effort into it.
However yesterday I was using Kapp.ai on the Discord forum to see what it could note and that lead to this
https://help.openai.com/en/articles/6516417-dall-e-editor-guide#h_04ec3d706f
which might be of value. However I don’t plan to try it at present but noting for others as I would here to hear if that is of value for the goal noted in the first post.
I have code that fairly reliably can take a prompt → GPT → high school physics diagrams for particular chapters/topics (mostly kinematics, because its just boxes). I will say that my use case is largely to make writing MCQ tests easier for teachers, not publishing textbooks. I am trying to expand this to biology (theoretically it should work for simple diagrams), but have other conflicting obligations. I would love advice and/or help if anyone is interested. Bonus points if you have friends at BioRender.
Use matplotlib to draw and label a heart
Planetary motion with hd diagram with all the notation
I am also a high school physics teacher trying to solve this problem. Would you be willing to share?
This is the number one feature of image generation tools I want to see. It baffles me how I am looking for something so simple and these tools that can solve complex math and summarize research level physics in seconds can’t reliably draw a block sliding down an inclined plane, let alone label the forces and spell five letter words properly in an image.
While image generation has dramatically improved over the last few years, the images from LLMs are still very dependent upon the LLM or model used. While I have not used commercial tools for image creation augmented with AI (think Adobe Firefly ), I would expect them to be better than those not specifically designed to do so .
For images of scientific quality, I find that having the images created as SVG (Scaled Vector Graphics) works better. While they may not be as nice as the diffusion-generated images, the results are more predictable, easier to modify, and often can be used with coding tools, e.g. canvas, built into the popular LLMs .
Bonus:
For a good technical understanding of how AI creates images, this survey paper has a section that explains such. The paper is from 2024, so while it is good for a general understanding, the recent progress is missing.
Text -to-image Diffusion Models in Generative AI: A Survey (PDF)
NB
The reference to Adobe Firefly is not a recommendation or suggestion, it is to give an example of the set of such tools.
Update
The following paper which is not directly related to the question, goes into more detail about creating SVG images from a text prompt, and then getting the SVGs to have a common style.
As with many research papers, included are introductions to some of the fundamentals related to creating SVG images from text prompts.
Style Customization of Text-to-Vector Generation with Image Diffusion Priors (PDF)