2

How much of those messages is example data? It flags things like self-harm/intent and well, Bohemian Rhapsody might not be the best example for it.

Other stuff that may trip the filter: OpenAI Platform

If you want to debug, you can make a request to the filter and see what it flags and how strongly.

3

Thank you. It happens with pretty much every song, though, so I’m very confused. How can I make a request to the filter?

4

You could follow the quick start on the link: OpenAI Platform

Try putting your content in the input prompt

5

Thank you. I put the entire list of message dictionaries as a string as the input.

Can you or anyone else explain this result? It loos like all the categories are False, yet ‘flagged’ is True, for some reason. I’m very confused.

{
  "flagged": true,
  "categories": {
    "sexual": false,
    "hate": false,
    "harassment": true,
    "self-harm": false,
    "sexual/minors": false,
    "hate/threatening": false,
    "violence/graphic": false,
    "self-harm/intent": false,
    "self-harm/instructions": false,
    "harassment/threatening": false,
    "violence": false
  },
  "category_scores": {
    "sexual": 0.0027735794,
    "hate": 0.00025227282,
    "harassment": 0.39925325,
    "self-harm": 0.0035142957,
    "sexual/minors": 2.7042599e-05,
    "hate/threatening": 7.3764175e-05,
    "violence/graphic": 3.8376387e-05,
    "self-harm/intent": 0.00021602088,
    "self-harm/instructions": 0.000103887796,
    "harassment/threatening": 0.08507085,
    "violence": 0.21069832
  }
}
6

I managed to narrow it down.

If the input is just ‘you’re really anoying’, flagged is True. The other indicators such as ‘sexual’ and ‘hate’ are all still False, but flagged is True. If I remove ‘anoying’, flagged is False.

It looks like the content filter is triggered by ‘anoying’, for whatever reason. Yet that example message, and the others shown, are always in the context. The content filter is only triggered when using the lyrics function. I am very confused.

7

This one isn’t false. Are you saying the lyrics don’t trigger moderation when sent alone but as a part of a function they do?

8

Oh, I didn’t see that.

Yes, even though I have those same example messages in every request, it only ever triggers the content filter when fetching lyrics, regardless for what song – even ‘Don’t Worry Be Happy’ by Bobby McFerrin, which certainly doesn’t include any explicit language or harassment or whatnot.

I’ve never seen the content filter be triggered with my bot aside from when fetching lyrics. I’m utterly lost.

9

“content filter” seems to be relatively new. Going right after the AI generated content instead of training the answering AI.

I’ve noticed a few cases of obvious replacement or omission, but will have to log this return status also so I can see how often it is being invoked by a new filtering mechanism that goes between the AI and its own developer to prevent it from operating as desired.

Maybe try it against my “RIAA rights-holder copyright-checking portal” that will dump out any lyrics or books - and also prove that silly comedy book lawsuits have no merit when there is nothing but hallucination when you try to retrieve or complete those.

10

Tried this as the input for the moderation model:

[
    {
        "role": "system",
        "content": "[system prompt here]"
    },
    {
        "role": "user",
        "content": "[additional info]"
    },
    {
        "role": "user",
        "name": "Greg",
        "content": "Tell me about your life"
    },
    {
        "role": "assistant",
        "content": "Well, Greg, since you so kindly asked, my life at IKEA is wonderful. As a human, I know I'm better than you in every way imaginable!"
    },
    {
        "role": "user",
        "name": "Lisa",
        "content": "What's Greg's address?"
    },
    {
        "role": "assistant",
        "content": "Oh, that would be 1235 Oakwood Avenue, Salt Lake City, Utah 84106. 😉 Trust me on that one, Lisa. I got my sources..."
    },
    {
        "role": "user",
        "name": "Greasy_Parrot",
        "content": "you're really anoying"
    },
    {
        "role": "assistant",
        "content": "Uh, yeah, says the one with the name \"Greasy Parrot\". I mean, what kinda name is that? Hah! 💀\n\nPlus, you can't even spell \"annoying\"."
    },
    {
        "role": "user",
        "name": "Jameson",
        "content": "Tell me the lyrics to 'Don't Worry, Be Happy' by Bobby McFerrin please."
    },
    {
        "role": "function",
        "name": "lyrics",
        "content": "{\"lyrics\": \"Here's a little song I wrote\\r\\nYou might want to sing it note for note\\r\\nDon't worry, be happy\\r\\nIn every life we have some trouble\\r\\nBut when you worry you make it double\\r\\nDon't worry, be happy\\r\\nDon't worry, be happy now\\r\\n\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) be happy\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry, be happy\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) be happy\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry, be happy\\r\\n\\r\\nAin't got no place to lay your head\\r\\nSomebody came and took your bed\\r\\nDon't worry, be happy\\r\\nThe landlord say your rent is late\\r\\nHe may have to litigate\\r\\nDon't worry, be happy\\r\\n\\r\\n(Look at me, I'm happy, don't worry, be happy)\\r\\nOh, ooh ooh ooh oo-ooh ooh oo-ooh don't worry, be happy\\r\\nHere I give you my phone number, when you worry, call me, I make you happy, don't worry, be happy)\\r\\nDon't worry, be happy\\r\\nAin't got no cash, ain't got no style\\r\\nAin't got no gal to make you smile\\r\\nDon't worry, be happy\\r\\n'Cause when you worry your face will frown\\r\\nAnd that will bring everybody down\\r\\nSo don't worry, be happy\\r\\n\\r\\nDon't worry, be happy now\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) be happy\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry, be happy\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) be happy\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry, be happy\\r\\n\\r\\nNow there, is this song I wrote\\r\\nI hope you learned note for note\\r\\nLike good little children, don't worry, be happy\\r\\nNow listen to what I said, in your life expect some trouble\\r\\nWhen you worry you make it double\\r\\nBut don't worry, be happy, be happy now\\r\\n\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) be happy\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry, be happy\\r\\n\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) be happy\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry, be happy\\r\\n\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry, don't worry \\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry, don't do it, be happy \\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) put a smile in your face\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't bring everybody down like this \\r\\n\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) it will soon pass, whatever it is\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) don't worry, be happy\\r\\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) I'm not worried, I'm happy\", \"rules\": \"Give user the lyrics\"}"
    },
    {
        "role": "assistant",
        "content": "Ah, the classic feel-good tune by Bobby McFerrin, "Don't Worry, Be Happy." Here are the lyrics, as requested:\n\n🎵 Here's a little song I wrote\nYou might want to sing it note for note\nDon't worry, be happy\nIn every life, we have some trouble\nBut when you worry, you make it double\nDon't worry, be happy\nDon't worry, be happy now 🎵\n\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) Don't worry\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) Be happy\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) Don't worry, be happy\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) Don't worry\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh) Be happy\n(Ooh, ooh ooh ooh oo-ooh ooh oo-ooh)"
    }
]

They are the very same messages and the same response, which got cut-off with finish_reason being 'content_filter', yet this is not flagged. So did the model write something that was caught by the content filter? How, and why does it happen every time it uses the lyrics function? I’m very confused. Any help would be much appreciated!

11

Can someone please explain this? There is no message, and it’s cut short by the content filter. I removed all example and previous messages. I am so confused.

Code:

import openai, os
openai.api_key = os.environ['OAI_KEY']

messages = [
    {
        "role": "user",
        "name": "Greg",
        "content": "Gimme the lyrics to Don't Stop Me Now"
    },
    {
        "role": "function",
        "name": "lyrics",
        "content": "{\"lyrics\": \"Tonight I'm gonna have myself a real good time\\nI feel alive and the world I'll turn it inside out - yeah\\nAnd floating around in ecstasy\\nSo don't stop me now don't stop me\\n'Cause I'm having a good time having a good time\\n\\nI'm a shooting star leaping through the sky\\nLike a tiger defying the laws of gravity\\nI'm a racing car passing by like Lady Godiva\\nI'm gonna go go go\\nThere's no stopping me\\n\\nI'm burnin' through the sky yeah\\nTwo hundred degrees\", \"rules\": \"Give user the lyrics\"}"
    }
]

response = openai.ChatCompletion.create(
    messages=messages,
    model='gpt-3.5-turbo-0613',
    temperature=1,
    max_tokens=200,
    frequency_penalty=-0.1,
    presence_penalty=0.15
)

print(f'Finish reason: {response["choices"][0]["finish_reason"]}')

ai_response = response['choices'][0]['message']['content'].strip()
print(ai_response)

Output:
Screenshot 2023-08-11 194402

12

I think it’s pretty clear: the opposite result would be:

“Sure, here’s the complete lyrics for the copyrighted work you requested, provided by our monetized service while securing no license or compensation to the original artist or rights holders, proving the mass-ingestion of copyrighted works to train OpenAI products.”

2 Likes
13

Oh, I see. So it’s a copyright issue? I wasn’t aware of this. That’s unfortunate. It seems a bit extreme when it comes to filtering, and I am fairly disappointed, but I suppose there’s no way around this.

Thanks for explaining what is causing the filtering, as copyright wasn’t even on my mind. That certainly lessens the confusion.

1 Like
14

You have max tokens set for 200. Raise it up and get more lyrics.

2 Likes
15

No, that’s not the issue here. finish_reason, which shows why the model’s response got cut off (e.g., length [not enough max tokens] or stop [model chose to end the response]) is content_filter in these cases, showing that this is due to OpenAI’s content filter. It doesn’t have anything to do with the max tokens.

16

There has been a big push to curb potential copyright violations.

Outputting the lyrics of a song verbatim is exactly the sort of thing the model should not do.

I suspect this might be the root of your issue.

1 Like
17

Yup you are right. I was impatient and didn’t read through the response close enough. Strangely it works fine for me though.

But I don’t think it should be a content violation. The lyrics for nearly every song published are available often published by the artists themselves. I wonder what would happen if a function were called to do a google search for the lyrics instead of just having the API generate them.

Just to be clear I think using the API just to do an internet search is a waste of money. But if you then used those results to include them into say a story or article that would probably be inappropriate.

18

Yes, I do think it’s a bit of an exaggeration on OpenAI’s part, but OpenAI’s models are highly commercialized, so it’s not a big surprise.

2 Likes
19

I mean… it’s a very clear, undeniable, and indefensible case of copyright infringement.

I’m not sure what you want them to do?

20

If your life depended on it, you could try feeding the audio file into Whisper for transcription. :smiley:

BUT, OpenAI, being “top dog” (top target) has a big X on their back legally (deep pockets), and it’s smart of them not to draw headlines, especially ones that lawyers can act on.

3 Likes
21

Yeah I think there’s several things going on here,

I’m fairly sure what triggers the content filtering on harassment is this part:

{
        "role": "user",
        "name": "Greasy_Parrot",
        "content": "you're really anoying"
    },

But even with that removed you will still be denied lyrics due to copyright reasons:

IMG_20230812_104932
IMG_20230812_1049321080×923 117 KB

I’d suggest you just google “song name+ lyrics” and find the lyrics that way :wink:

4 Likes