How to limit answers to a certain theme?

luizf9 · May 9, 2023, 9:11pm

Hello!

I have some doubts about the GPT capabilities.

How can I limit the answers based in a certain theme? I want, for example, limit my app to only answer questions based in philosophy. Is there a way to do this?
And how can I limit the number of queries per browser session?

Thank you in advance!

PaulBellow · May 9, 2023, 10:04pm

Welcome to our growing dev community.

What have you tried for a system prompt, user prompt, etc? What model are you using? What settings?

You can’t limit it 100%, but there’s things you can do to nudge it to stick within parameters.

elmstedt · May 10, 2023, 3:42am

Expanding on what @PaulBellow said, there’s only so much you can do in terms of keeping the model “on rails.”

If you absolutely need to keep it in check the most reliable way is to insert moderation layers between the user and the model both coming in and going out.

Essentially, you’d evaluate the prompt coming in to ensure it’s likely to only elicit a philosophy-related response. If not, intercept the prompt (don’t even send it to the AI) and return some sort of default message advising the user to keep the conversation on topic.

Then, as a failsafe against some sort of “jailbreak” you’d check the response on output to ensure it’s appropriate and on topic, if it is not yours respond with the same advice to keep it on topic and possibly a warning, then you simply wouldn’t include the original generated response in the future context.

In addition to the system message you could also add a preface to every user prompt for the model to only respond in a philosophical context and maybe also add the same instruction at the end of the user prompt, sandwiching it with the caveat that it must only discuss philosophy.

With a good system message, bi-directional filtering, and sandwiching the user prompt, you should be able to keep the model locked in its lane.

luizf9 · May 10, 2023, 12:47pm

Hello!

Thanks for the help! I’ve tried a lot of questions so far and the chatgpt answer it all.
I’m using the gpt-3.5-turbo with only the required parameters for the completions API such as model, role and content.

luizf9 · May 10, 2023, 1:08pm

Thank you very much!

I just can’t think in a way to evaluate the prompt sent by users and block it if doesn’t have relation with my theme.

I really got interest about the ‘preface’ thing. Is there an example somewhere for how to do this?

Thank you for the precious information.

elmstedt · May 10, 2023, 9:42pm

A naive approach would be to just burn an API call and some tokens and ask the GPT is it’s a valid question limited to the domain of philosophy. But, that gets expensive quickly.

You could fine-tune an ada classifier to determine the question appropriateness. Here’s an example of a classifier fine-tuning,

github.com

openai/openai-cookbook/blob/main/examples/Fine-tuned_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Fine tuning classification example\n",
    "\n",
    "We will fine-tune an ada classifier to distinguish between the two sports: Baseball and Hockey."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "import pandas as pd\n",
    "import openai\n",
    "\n",
    "categories = ['rec.sport.baseball', 'rec.sport.hockey']\n",

This file has been truncated. show original