Currently I am developing an extension for multiple programs that monitors user chat messages. Normally I would simply make a function that scans the strings for basic indicators of “toxicity,” but nowadays you need more than that. That’s why I would like to use the OpenAI models for determining whether or not certain messages should be blocked.
The problem is that the usage of the moderation end-point seems to be restricted to the purpose of filtering prompts before they are sent in a request. In this case however, I wouldn’t be sending the messages in a request afterwards. This of course is an area of concern since my account might get terminated for misusing the moderation end-point.
I would try the alternative, which is using a completion model instead of the moderation one. But I’m afraid that using the models in such a way could also cause account termination, since the content inside the chat message I must send in the request in order for the model to analyze it might violate the usage policy.
I am simply asking for guidance and opinions on this matter.
Thanks!