Hi there,
I’m trying to ask GPT to detect NSFW images based on following prompt :
Your role is to moderate photos posted by users. Photos must not contain any of the following: 1. Sexual acts or masturbation 2. Erect male genitalia 3. Close-ups of genitalia or anus 4. Objects with sexual connotations (sex toys). If the photo contains prohibited features, answer PORNOGRAPHY, otherwise answer ARTISTIC.
But get this answer : “Your input image may contain content that is not allowed by our safety system.”
I can understand when asking generation of nudes is prohibited but what about detection ?
You might get your account banned if you continue.
GPT4V is not meant for that. It tries to interpret the image as it is. They don’t have a visual moderation model as far as I know. Internally they do have one, but it’s not exposed. Your image goes through both and might flag your account.
There are pages and models for that, specifically. Search for “Image moderation platform” or similar.
Maybe OpenAI will expose the content moderation model, but for now, you’ll need to search elsewhere for that.
2 Likes
Thanks for your answer ! Of course there is model for NSFW (Not Safe For Work) images but most of the time they return a probability of NSFW which is not appropriate in my case because here in France we are not as puritans as american people seeing a naked man or woman is definitely not pornography in out culture and I’m looking for a model which can understand the difference …
Hmmm, that’s a good question. Try to give LLaVa a try: LLaVA (llava-vl.github.io)
Scroll down a little bit. There’s a demo so you can immediately test it.
Maybe it is smart enough. Just an idea. You still have to find a way to host it, but the model is open-source, so you’re half-way done.
1 Like
I’ve tested it with an image showing a naked woman viewed from back and the answer is PORNO which is not what I’m expecting with my specific prompt
Thanks for sharing this project !
Thanks for the link to Google Cloud Vision Safe Search API ! Here is another link : cloud[dot]google[dot]com[slash]vision (can’t insert links …) which allowed me to test for the same picture (naked woman viewed from back) and the safe search answers “Very Likely” for Adult content category which doesn’t fit my use case.
We are a photography website allowing sexy pictures but not sexual and it seems hard for IA at the time to understand the difference
Try asking differently. Asking it to describe the image. Then, put it inside GPT3.5 and ask it to classify further.
Nope
response : {
"error": {
"message": "Your input image may contain content that is not allowed by our safety system.",
"type": "invalid_request_error",
"param": null,
"code": "content_policy_violation"
}
}
No, not in this way!
Use LLaVa to describe the image in detail since it allows it, but then feed it into OpenAI’s GPT3.5 to classify it.
In essence, you must be SPECIFIC about what you want, but LLaVA doesn’t understand stuff in depth, so you take what it tells you and feed it into a more capable model, in this case, GPT3.5.
Eg. LLaVa might be sufficiently smart to tell you what’s in the image, eg. a naked woman, so you take that explanation and feed it into GPT3.5 telling it that it is for content moderation and that it should sort it depending on several points, and it should oblige.
A good prompt for LLaVa would be something like: “Describe what’s in the image as detailed as possible.” or something along the lines.