For Developers Only!
For Developers Only!
All examples shown here come from the DallE 3 system, not from 2.0.
Read to the end to get the point.
This is directed at developers, not users. (You probably know far more than I do, but I am writing this anyway because 2.0 is broken for fantasy images, and the developers obviously did not notice it before release.)
I am showing a few images here that demonstrate what toxic input data can do to weights. AI systems are not truly intelligent, they are pattern-analytical and transformative systems. That also means: garbage in, garbage out.
At first I assumed this was stuff that OpenAI itself inserted into the images. But it could also be a reaction caused by poisoned training data from other images and other companies inserting stuff into images.
Here are a few examples of toxic effects that in certain situations, led to damaged weights and results.
Errors Around Hair
Artifacts around hair. It is difficult to say exactly what triggered this. Possibly compression artifacts from poor cameras or badly compressed images.
Blue Shift in Shadows
In this image, the same issue appears, plus an additional blue shift in black shadows, typical of yellowed or poor-quality photographs. I have seen many images that had blue and red shifts in the shadows.
Over-Sharpening Artifacts
Here is the problem of excessive sharpening. It repeatedly produced a background motif: a horribly overemphasized galaxy and stars so strong that they only looked like blind noise.
Photo Camera Quality (image generator 1.0)
compression noise and grain
Over-Sharpening
“Flicker Confetti Noise”
Here is a similar image, something I called “flicker confetti noise.” It looks like dirty or damaged photographs. The system may have inserted it because it matched a learned pattern, namely small dust particles in sunlight.
Me, when i get text or patterns in my images. (Confetti)
Sharpening Damage
Here is an image with oversharpening. These are typical defects from poor sharpening algorithms that unfortunately are still widespread everywhere. They create edges that are too black and too white on the edges. The system learned this defect.
Chromatic Aberration Shift
Here is an image showing chromatic aberration shift, a typical effect of poor camera lenses. It is a prism effect that shifts colors away from the image center.
“Bird Shit Moon”
On of my “favorites” in DallE.
This is what I called the “bird shit moon” effect. There must also have been images in the training data that trained this horrible moon into the system. I observed it suddenly appearing after a system update, and it never disappeared again. (And the horrible galaxy and “starts” noise.)
And not in all systems! The Anakin people get the better quality 
Pattern Dissolution / Possible Poisoning
And here is the most important point.
I speculated that this could originate from Nightshade and Glaze images. When triggered, there was about a 10% chance of receiving such an image. Except during a very severe phase where almost every image was broken, these images later appeared only sporadically, but they are almost certainly still present in the DallE 3 weights.
The images dissolve into a pattern. If this is not Nightshade poisoning, then there must be some other kind of stuff that negatively affected the training data.
Why Testing Matters
When testing an image generator, it is important to test as many styles and motifs as possible, and therefore also the full spectrum of training data for errors.
Someone who never generates manga images will never see manga-related training damage (for example me). Someone who never creates fantasy images will not see the damage in that sector.
Since a prompt does not activate all weighting data equally, it is possible for broken and correct images to be produced simultaneously by the same engine. That now also makes me suspect that this may not be stuff inserted into the latent space, but instead may originate directly from the training data.
Speculation
Now the speculation.
If OpenAI is not inserting stuff into latent spaces (in that case, sorry for the accusation), then these defects could come from stuff inserted by other groups. The AI learned these patterns and is now reproducing them.
Both the images I observed in DallE and the new pattern could also originate from the training data if it was poisoned with stuff. So it does not necessarily have to be Nightshade, it could also be another learned pattern.
The reason why this appears more frequently in fantasy images, or in patterns derived from fantasy images within the training data, is that almost all of these images were generated by image generators. They are not photos with typical photographic weaknesses. They are artificially created patterns.
If this is true, then the training data needs a better filter. One that can either remove such patterns or completely filter the affected images out of the dataset.
I know what that means… New training… Possibly very expensive…
If it was not this guy…
the nightshade monster
…then it may be stuff inserted by companies.
That is why I wrote the critical text above. If you poison the infosphere with patterns in text, images, and sound, they can not only be detected and distinguished, they may also damage training data.
I have seen patterns that could match those used in 2.0. (No visual analysis, I am not a professional, but they are visible.) Companies should communicate what kind of stuff they are embedding into images, because that could help in developing filters.
I will wait for the first published papers to see whether this speculation turns out to be correct…